Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2838 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.4872. Min Range: -0.4872

Max Act: 0.4638. Min Act: -0.0000

Data Index: 1435812 (Python Code)

Max Activating Token Index: 320

Text #1

Max Range: 0.4872. Min Range: -0.4872

Max Act: 0.4872. Min Act: -0.0000

Data Index: 1536440 (Python Code)

Max Activating Token Index: 207

Text #2

Max Range: 0.4872. Min Range: -0.4872

Max Act: 0.4414. Min Act: -0.0000

Data Index: 1650785 (Python Code)

Max Activating Token Index: 550

Text #3

Max Range: 0.4872. Min Range: -0.4872

Max Act: 0.4510. Min Act: -0.0000

Data Index: 1492744 (Python Code)

Max Activating Token Index: 370

Text #4

Max Range: 0.4872. Min Range: -0.4872

Max Act: 0.4176. Min Act: -0.0000

Data Index: 1507143 (Python Code)

Max Activating Token Index: 475

Text #5

Max Range: 0.4872. Min Range: -0.4872

Max Act: 0.4402. Min Act: -0.0000

Data Index: 1360357 (Python Code)

Max Activating Token Index: 501

Text #6

Max Range: 0.4872. Min Range: -0.4872

Max Act: 0.4462. Min Act: -0.0000

Data Index: 1486982 (Python Code)

Max Activating Token Index: 28

Text #7

Max Range: 0.4872. Min Range: -0.4872

Max Act: 0.4372. Min Act: -0.0000

Data Index: 1440605 (Python Code)

Max Activating Token Index: 1009

Text #8

Max Range: 0.4872. Min Range: -0.4872

Max Act: 0.4232. Min Act: -0.0000

Data Index: 1590050 (Python Code)

Max Activating Token Index: 689

Text #9

Max Range: 0.4872. Min Range: -0.4872

Max Act: 0.4197. Min Act: -0.0000

Data Index: 1548486 (Python Code)

Max Activating Token Index: 536

Text #10

Max Range: 0.4872. Min Range: -0.4872

Max Act: 0.4060. Min Act: -0.0000

Data Index: 1479685 (Python Code)

Max Activating Token Index: 258

Text #11

Max Range: 0.4872. Min Range: -0.4872

Max Act: 0.3667. Min Act: -0.0000

Data Index: 1456815 (Python Code)

Max Activating Token Index: 987

Text #12

Max Range: 0.4872. Min Range: -0.4872

Max Act: 0.3883. Min Act: -0.0000

Data Index: 1556329 (Python Code)

Max Activating Token Index: 1017

Text #13

Max Range: 0.4872. Min Range: -0.4872

Max Act: 0.3975. Min Act: -0.0000

Data Index: 1397970 (Python Code)

Max Activating Token Index: 112

Text #14

Max Range: 0.4872. Min Range: -0.4872

Max Act: 0.4038. Min Act: -0.0000

Data Index: 1559148 (Python Code)

Max Activating Token Index: 936

Text #15

Max Range: 0.4872. Min Range: -0.4872

Max Act: 0.3999. Min Act: -0.0000

Data Index: 1575049 (Python Code)

Max Activating Token Index: 1021

Text #16

Max Range: 0.4872. Min Range: -0.4872

Max Act: 0.3896. Min Act: -0.0000

Data Index: 1503662 (Python Code)

Max Activating Token Index: 171

Text #17

Max Range: 0.4872. Min Range: -0.4872

Max Act: 0.4118. Min Act: -0.0000

Data Index: 1578721 (Python Code)

Max Activating Token Index: 374

Text #18

Max Range: 0.4872. Min Range: -0.4872

Max Act: 0.3710. Min Act: -0.0000

Data Index: 1488803 (Python Code)

Max Activating Token Index: 992

Text #19

Max Range: 0.4872. Min Range: -0.4872

Max Act: 0.4091. Min Act: -0.0000

Data Index: 1383308 (Python Code)

Max Activating Token Index: 937