Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2298 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.6064. Min Range: -0.6064

Max Act: 0.6064. Min Act: -0.0000

Data Index: 1030021 (C4 (Web Text))

Max Activating Token Index: 877

Text #1

Max Range: 0.6064. Min Range: -0.6064

Max Act: 0.5688. Min Act: -0.0000

Data Index: 1070604 (C4 (Web Text))

Max Activating Token Index: 300

Text #2

Max Range: 0.6064. Min Range: -0.6064

Max Act: 0.5220. Min Act: -0.0000

Data Index: 1041579 (C4 (Web Text))

Max Activating Token Index: 323

Text #3

Max Range: 0.6064. Min Range: -0.6064

Max Act: 0.4852. Min Act: -0.0000

Data Index: 755586 (C4 (Web Text))

Max Activating Token Index: 763

Text #4

Max Range: 0.6064. Min Range: -0.6064

Max Act: 0.5143. Min Act: -0.0000

Data Index: 445967 (C4 (Web Text))

Max Activating Token Index: 1011

Text #5

Max Range: 0.6064. Min Range: -0.6064

Max Act: 0.5186. Min Act: -0.0000

Data Index: 176176 (C4 (Web Text))

Max Activating Token Index: 774

Text #6

Max Range: 0.6064. Min Range: -0.6064

Max Act: 0.4956. Min Act: -0.0000

Data Index: 434028 (C4 (Web Text))

Max Activating Token Index: 923

Text #7

Max Range: 0.6064. Min Range: -0.6064

Max Act: 0.4363. Min Act: -0.0000

Data Index: 557989 (C4 (Web Text))

Max Activating Token Index: 182

Text #8

Max Range: 0.6064. Min Range: -0.6064

Max Act: 0.4612. Min Act: -0.0000

Data Index: 1213120 (C4 (Web Text))

Max Activating Token Index: 31

Text #9

Max Range: 0.6064. Min Range: -0.6064

Max Act: 0.4340. Min Act: -0.0000

Data Index: 194148 (C4 (Web Text))

Max Activating Token Index: 488

Text #10

Max Range: 0.6064. Min Range: -0.6064

Max Act: 0.4501. Min Act: -0.0000

Data Index: 1336188 (C4 (Web Text))

Max Activating Token Index: 188

Text #11

Max Range: 0.6064. Min Range: -0.6064

Max Act: 0.4378. Min Act: -0.0000

Data Index: 1029802 (C4 (Web Text))

Max Activating Token Index: 964

Text #12

Max Range: 0.6064. Min Range: -0.6064

Max Act: 0.4337. Min Act: -0.0000

Data Index: 716381 (C4 (Web Text))

Max Activating Token Index: 691

Text #13

Max Range: 0.6064. Min Range: -0.6064

Max Act: 0.4056. Min Act: -0.0000

Data Index: 1276368 (C4 (Web Text))

Max Activating Token Index: 883

Text #14

Max Range: 0.6064. Min Range: -0.6064

Max Act: 0.4228. Min Act: -0.0000

Data Index: 463074 (C4 (Web Text))

Max Activating Token Index: 232

Text #15

Max Range: 0.6064. Min Range: -0.6064

Max Act: 0.4427. Min Act: -0.0000

Data Index: 702178 (C4 (Web Text))

Max Activating Token Index: 410

Text #16

Max Range: 0.6064. Min Range: -0.6064

Max Act: 0.4259. Min Act: -0.0000

Data Index: 380863 (C4 (Web Text))

Max Activating Token Index: 187

Text #17

Max Range: 0.6064. Min Range: -0.6064

Max Act: 0.3930. Min Act: -0.0000

Data Index: 943846 (C4 (Web Text))

Max Activating Token Index: 504

Text #18

Max Range: 0.6064. Min Range: -0.6064

Max Act: 0.4340. Min Act: -0.0000

Data Index: 902907 (C4 (Web Text))

Max Activating Token Index: 739

Text #19

Max Range: 0.6064. Min Range: -0.6064

Max Act: 0.4232. Min Act: -0.0000

Data Index: 422586 (C4 (Web Text))

Max Activating Token Index: 532