Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2043 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 2.6282. Min Range: -2.6282

Max Act: 2.6282. Min Act: -0.0000

Data Index: 73167 (C4 (Web Text))

Max Activating Token Index: 832

Text #1

Max Range: 2.6282. Min Range: -2.6282

Max Act: 2.1109. Min Act: -0.0000

Data Index: 866926 (C4 (Web Text))

Max Activating Token Index: 475

Text #2

Max Range: 2.6282. Min Range: -2.6282

Max Act: 1.8895. Min Act: -0.0000

Data Index: 1340082 (C4 (Web Text))

Max Activating Token Index: 413

Text #3

Max Range: 2.6282. Min Range: -2.6282

Max Act: 2.0054. Min Act: -0.0000

Data Index: 1016642 (C4 (Web Text))

Max Activating Token Index: 913

Text #4

Max Range: 2.6282. Min Range: -2.6282

Max Act: 1.8791. Min Act: -0.0000

Data Index: 655670 (C4 (Web Text))

Max Activating Token Index: 221

Text #5

Max Range: 2.6282. Min Range: -2.6282

Max Act: 1.9587. Min Act: -0.0000

Data Index: 536645 (C4 (Web Text))

Max Activating Token Index: 160

Text #6

Max Range: 2.6282. Min Range: -2.6282

Max Act: 1.6879. Min Act: -0.0000

Data Index: 919977 (C4 (Web Text))

Max Activating Token Index: 250

Text #7

Max Range: 2.6282. Min Range: -2.6282

Max Act: 1.7664. Min Act: -0.0000

Data Index: 2861 (C4 (Web Text))

Max Activating Token Index: 189

Text #8

Max Range: 2.6282. Min Range: -2.6282

Max Act: 1.9560. Min Act: -0.0000

Data Index: 287790 (C4 (Web Text))

Max Activating Token Index: 858

Text #9

Max Range: 2.6282. Min Range: -2.6282

Max Act: 1.8462. Min Act: -0.0000

Data Index: 1276598 (C4 (Web Text))

Max Activating Token Index: 574

Text #10

Max Range: 2.6282. Min Range: -2.6282

Max Act: 1.9209. Min Act: -0.0000

Data Index: 93219 (C4 (Web Text))

Max Activating Token Index: 994

Text #11

Max Range: 2.6282. Min Range: -2.6282

Max Act: 1.8029. Min Act: -0.0000

Data Index: 1317765 (C4 (Web Text))

Max Activating Token Index: 523

Text #12

Max Range: 2.6282. Min Range: -2.6282

Max Act: 1.7581. Min Act: -0.0000

Data Index: 1314495 (C4 (Web Text))

Max Activating Token Index: 569

Text #13

Max Range: 2.6282. Min Range: -2.6282

Max Act: 1.7385. Min Act: -0.0000

Data Index: 660334 (C4 (Web Text))

Max Activating Token Index: 55

Text #14

Max Range: 2.6282. Min Range: -2.6282

Max Act: 1.6350. Min Act: -0.0000

Data Index: 592833 (C4 (Web Text))

Max Activating Token Index: 881

Text #15

Max Range: 2.6282. Min Range: -2.6282

Max Act: 1.6730. Min Act: -0.0000

Data Index: 74160 (C4 (Web Text))

Max Activating Token Index: 791

Text #16

Max Range: 2.6282. Min Range: -2.6282

Max Act: 1.6417. Min Act: -0.0000

Data Index: 1108912 (C4 (Web Text))

Max Activating Token Index: 896

Text #17

Max Range: 2.6282. Min Range: -2.6282

Max Act: 1.5516. Min Act: -0.0000

Data Index: 1240526 (C4 (Web Text))

Max Activating Token Index: 198

Text #18

Max Range: 2.6282. Min Range: -2.6282

Max Act: 1.6333. Min Act: -0.0000

Data Index: 1207927 (C4 (Web Text))

Max Activating Token Index: 347

Text #19

Max Range: 2.6282. Min Range: -2.6282

Max Act: 1.5319. Min Act: -0.0000

Data Index: 256507 (C4 (Web Text))

Max Activating Token Index: 712