Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1920 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.1271. Min Range: -1.1271

Max Act: 1.1271. Min Act: -0.0000

Data Index: 1408697 (Python Code)

Max Activating Token Index: 272

Text #1

Max Range: 1.1271. Min Range: -1.1271

Max Act: 1.0425. Min Act: -0.0000

Data Index: 1393274 (Python Code)

Max Activating Token Index: 20

Text #2

Max Range: 1.1271. Min Range: -1.1271

Max Act: 0.9992. Min Act: -0.0000

Data Index: 1508927 (Python Code)

Max Activating Token Index: 159

Text #3

Max Range: 1.1271. Min Range: -1.1271

Max Act: 0.9571. Min Act: -0.0000

Data Index: 1471421 (Python Code)

Max Activating Token Index: 19

Text #4

Max Range: 1.1271. Min Range: -1.1271

Max Act: 1.0179. Min Act: 0.0000

Data Index: 1502279 (Python Code)

Max Activating Token Index: 122

Text #5

Max Range: 1.1271. Min Range: -1.1271

Max Act: 0.9640. Min Act: -0.0000

Data Index: 1486902 (Python Code)

Max Activating Token Index: 12

Text #6

Max Range: 1.1271. Min Range: -1.1271

Max Act: 0.9874. Min Act: -0.0000

Data Index: 1385966 (Python Code)

Max Activating Token Index: 41

Text #7

Max Range: 1.1271. Min Range: -1.1271

Max Act: 0.9905. Min Act: -0.0000

Data Index: 1449198 (Python Code)

Max Activating Token Index: 90

Text #8

Max Range: 1.1271. Min Range: -1.1271

Max Act: 0.9385. Min Act: -0.0000

Data Index: 1589406 (Python Code)

Max Activating Token Index: 148

Text #9

Max Range: 1.1271. Min Range: -1.1271

Max Act: 0.8563. Min Act: -0.0000

Data Index: 1533524 (Python Code)

Max Activating Token Index: 96

Text #10

Max Range: 1.1271. Min Range: -1.1271

Max Act: 0.8537. Min Act: -0.0000

Data Index: 1412470 (Python Code)

Max Activating Token Index: 877

Text #11

Max Range: 1.1271. Min Range: -1.1271

Max Act: 0.8231. Min Act: 0.0000

Data Index: 1387625 (Python Code)

Max Activating Token Index: 405

Text #12

Max Range: 1.1271. Min Range: -1.1271

Max Act: 0.8878. Min Act: -0.0000

Data Index: 1555360 (Python Code)

Max Activating Token Index: 737

Text #13

Max Range: 1.1271. Min Range: -1.1271

Max Act: 0.7938. Min Act: -0.0000

Data Index: 1585977 (Python Code)

Max Activating Token Index: 177

Text #14

Max Range: 1.1271. Min Range: -1.1271

Max Act: 0.8202. Min Act: -0.0000

Data Index: 1403963 (Python Code)

Max Activating Token Index: 420

Text #15

Max Range: 1.1271. Min Range: -1.1271

Max Act: 0.8366. Min Act: -0.0000

Data Index: 1573320 (Python Code)

Max Activating Token Index: 22

Text #16

Max Range: 1.1271. Min Range: -1.1271

Max Act: 0.7833. Min Act: -0.0000

Data Index: 1510662 (Python Code)

Max Activating Token Index: 647

Text #17

Max Range: 1.1271. Min Range: -1.1271

Max Act: 0.8158. Min Act: -0.0000

Data Index: 1617208 (Python Code)

Max Activating Token Index: 761

Text #18

Max Range: 1.1271. Min Range: -1.1271

Max Act: 0.7947. Min Act: -0.0000

Data Index: 1556258 (Python Code)

Max Activating Token Index: 532

Text #19

Max Range: 1.1271. Min Range: -1.1271

Max Act: 0.7739. Min Act: -0.0000

Data Index: 1530624 (Python Code)

Max Activating Token Index: 30