Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 669 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.9380. Min Range: -1.9380

Max Act: 1.9380. Min Act: -0.0001

Data Index: 216832 (C4 (Web Text))

Max Activating Token Index: 790

Text #1

Max Range: 1.9380. Min Range: -1.9380

Max Act: 1.8245. Min Act: -0.0001

Data Index: 1240471 (C4 (Web Text))

Max Activating Token Index: 804

Text #2

Max Range: 1.9380. Min Range: -1.9380

Max Act: 1.3669. Min Act: -0.0001

Data Index: 748453 (C4 (Web Text))

Max Activating Token Index: 1020

Text #3

Max Range: 1.9380. Min Range: -1.9380

Max Act: 1.4497. Min Act: -0.0000

Data Index: 718668 (C4 (Web Text))

Max Activating Token Index: 970

Text #4

Max Range: 1.9380. Min Range: -1.9380

Max Act: 1.5354. Min Act: -0.0001

Data Index: 674402 (C4 (Web Text))

Max Activating Token Index: 183

Text #5

Max Range: 1.9380. Min Range: -1.9380

Max Act: 1.4435. Min Act: -0.0001

Data Index: 244864 (C4 (Web Text))

Max Activating Token Index: 197

Text #6

Max Range: 1.9380. Min Range: -1.9380

Max Act: 1.4293. Min Act: -0.0001

Data Index: 92679 (C4 (Web Text))

Max Activating Token Index: 259

Text #7

Max Range: 1.9380. Min Range: -1.9380

Max Act: 1.4632. Min Act: -0.0000

Data Index: 444224 (C4 (Web Text))

Max Activating Token Index: 19

Text #8

Max Range: 1.9380. Min Range: -1.9380

Max Act: 1.4833. Min Act: -0.0000

Data Index: 1301137 (C4 (Web Text))

Max Activating Token Index: 899

Text #9

Max Range: 1.9380. Min Range: -1.9380

Max Act: 1.4235. Min Act: -0.0001

Data Index: 312174 (C4 (Web Text))

Max Activating Token Index: 162

Text #10

Max Range: 1.9380. Min Range: -1.9380

Max Act: 1.3077. Min Act: -0.0001

Data Index: 962902 (C4 (Web Text))

Max Activating Token Index: 740

Text #11

Max Range: 1.9380. Min Range: -1.9380

Max Act: 1.3493. Min Act: -0.0001

Data Index: 380903 (C4 (Web Text))

Max Activating Token Index: 882

Text #12

Max Range: 1.9380. Min Range: -1.9380

Max Act: 1.3948. Min Act: -0.0001

Data Index: 319523 (C4 (Web Text))

Max Activating Token Index: 828

Text #13

Max Range: 1.9380. Min Range: -1.9380

Max Act: 1.3174. Min Act: -0.0000

Data Index: 882337 (C4 (Web Text))

Max Activating Token Index: 367

Text #14

Max Range: 1.9380. Min Range: -1.9380

Max Act: 1.3740. Min Act: -0.0000

Data Index: 993517 (C4 (Web Text))

Max Activating Token Index: 773

Text #15

Max Range: 1.9380. Min Range: -1.9380

Max Act: 1.3044. Min Act: -0.0000

Data Index: 807027 (C4 (Web Text))

Max Activating Token Index: 725

Text #16

Max Range: 1.9380. Min Range: -1.9380

Max Act: 1.3340. Min Act: -0.0001

Data Index: 916174 (C4 (Web Text))

Max Activating Token Index: 107

Text #17

Max Range: 1.9380. Min Range: -1.9380

Max Act: 1.3504. Min Act: -0.0000

Data Index: 325387 (C4 (Web Text))

Max Activating Token Index: 330

Text #18

Max Range: 1.9380. Min Range: -1.9380

Max Act: 1.3538. Min Act: -0.0000

Data Index: 888962 (C4 (Web Text))

Max Activating Token Index: 733

Text #19

Max Range: 1.9380. Min Range: -1.9380

Max Act: 1.3670. Min Act: -0.0001

Data Index: 440601 (C4 (Web Text))

Max Activating Token Index: 1010