Model: SoLU Model: 2 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 777 in Layer 0

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-2l')

Text #0

Max Range: 0.0345. Min Range: -0.0345

Max Act: 0.0345. Min Act: -0.0004

Data Index: 1398109 (Python Code)

Max Activating Token Index: 337

Text #1

Max Range: 0.0345. Min Range: -0.0345

Max Act: 0.0333. Min Act: -0.0004

Data Index: 1557300 (Python Code)

Max Activating Token Index: 179

Text #2

Max Range: 0.0345. Min Range: -0.0345

Max Act: 0.0330. Min Act: -0.0004

Data Index: 1616299 (Python Code)

Max Activating Token Index: 867

Text #3

Max Range: 0.0345. Min Range: -0.0345

Max Act: 0.0324. Min Act: -0.0004

Data Index: 1380311 (Python Code)

Max Activating Token Index: 888

Text #4

Max Range: 0.0345. Min Range: -0.0345

Max Act: 0.0321. Min Act: -0.0004

Data Index: 1523300 (Python Code)

Max Activating Token Index: 19

Text #5

Max Range: 0.0345. Min Range: -0.0345

Max Act: 0.0320. Min Act: -0.0004

Data Index: 1595223 (Python Code)

Max Activating Token Index: 692

Text #6

Max Range: 0.0345. Min Range: -0.0345

Max Act: 0.0312. Min Act: -0.0004

Data Index: 1409832 (Python Code)

Max Activating Token Index: 784

Text #7

Max Range: 0.0345. Min Range: -0.0345

Max Act: 0.0317. Min Act: -0.0004

Data Index: 1512441 (Python Code)

Max Activating Token Index: 648

Text #8

Max Range: 0.0345. Min Range: -0.0345

Max Act: 0.0309. Min Act: -0.0004

Data Index: 1489775 (Python Code)

Max Activating Token Index: 95

Text #9

Max Range: 0.0345. Min Range: -0.0345

Max Act: 0.0316. Min Act: -0.0004

Data Index: 1362843 (Python Code)

Max Activating Token Index: 568

Text #10

Max Range: 0.0345. Min Range: -0.0345

Max Act: 0.0323. Min Act: -0.0004

Data Index: 1653269 (Python Code)

Max Activating Token Index: 662

Text #11

Max Range: 0.0345. Min Range: -0.0345

Max Act: 0.0316. Min Act: -0.0004

Data Index: 1448665 (Python Code)

Max Activating Token Index: 334

Text #12

Max Range: 0.0345. Min Range: -0.0345

Max Act: 0.0308. Min Act: -0.0004

Data Index: 1435965 (Python Code)

Max Activating Token Index: 870

Text #13

Max Range: 0.0345. Min Range: -0.0345

Max Act: 0.0314. Min Act: -0.0004

Data Index: 1485096 (Python Code)

Max Activating Token Index: 15

Text #14

Max Range: 0.0345. Min Range: -0.0345

Max Act: 0.0307. Min Act: -0.0004

Data Index: 1439776 (Python Code)

Max Activating Token Index: 729

Text #15

Max Range: 0.0345. Min Range: -0.0345

Max Act: 0.0313. Min Act: -0.0004

Data Index: 1409521 (Python Code)

Max Activating Token Index: 870

Text #16

Max Range: 0.0345. Min Range: -0.0345

Max Act: 0.0312. Min Act: -0.0004

Data Index: 1623456 (Python Code)

Max Activating Token Index: 17

Text #17

Max Range: 0.0345. Min Range: -0.0345

Max Act: 0.0312. Min Act: -0.0004

Data Index: 1582631 (Python Code)

Max Activating Token Index: 816

Text #18

Max Range: 0.0345. Min Range: -0.0345

Max Act: 0.0305. Min Act: -0.0004

Data Index: 1613119 (Python Code)

Max Activating Token Index: 837

Text #19

Max Range: 0.0345. Min Range: -0.0345

Max Act: 0.0310. Min Act: -0.0004

Data Index: 1529711 (Python Code)

Max Activating Token Index: 648