Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4396 in Layer 10

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 2.8456. Min Range: -2.8456

Max Act: 2.8456. Min Act: -0.0000

Data Index: 1067901 (C4 (Web Text))

Max Activating Token Index: 728

Text #1

Max Range: 2.8456. Min Range: -2.8456

Max Act: 2.7188. Min Act: -0.0000

Data Index: 927232 (C4 (Web Text))

Max Activating Token Index: 701

Text #2

Max Range: 2.8456. Min Range: -2.8456

Max Act: 2.5336. Min Act: -0.0000

Data Index: 211208 (C4 (Web Text))

Max Activating Token Index: 901

Text #3

Max Range: 2.8456. Min Range: -2.8456

Max Act: 2.5120. Min Act: -0.0000

Data Index: 419918 (C4 (Web Text))

Max Activating Token Index: 124

Text #4

Max Range: 2.8456. Min Range: -2.8456

Max Act: 2.4372. Min Act: -0.0000

Data Index: 114062 (C4 (Web Text))

Max Activating Token Index: 242

Text #5

Max Range: 2.8456. Min Range: -2.8456

Max Act: 2.3257. Min Act: -0.0000

Data Index: 567164 (C4 (Web Text))

Max Activating Token Index: 912

Text #6

Max Range: 2.8456. Min Range: -2.8456

Max Act: 2.2716. Min Act: -0.0000

Data Index: 640896 (C4 (Web Text))

Max Activating Token Index: 1023

Text #7

Max Range: 2.8456. Min Range: -2.8456

Max Act: 2.2857. Min Act: -0.0000

Data Index: 1387 (C4 (Web Text))

Max Activating Token Index: 257

Text #8

Max Range: 2.8456. Min Range: -2.8456

Max Act: 2.2974. Min Act: -0.0000

Data Index: 1254034 (C4 (Web Text))

Max Activating Token Index: 780

Text #9

Max Range: 2.8456. Min Range: -2.8456

Max Act: 2.2641. Min Act: -0.0000

Data Index: 638571 (C4 (Web Text))

Max Activating Token Index: 569

Text #10

Max Range: 2.8456. Min Range: -2.8456

Max Act: 2.2106. Min Act: -0.0000

Data Index: 1174414 (C4 (Web Text))

Max Activating Token Index: 466

Text #11

Max Range: 2.8456. Min Range: -2.8456

Max Act: 2.2165. Min Act: -0.0000

Data Index: 657877 (C4 (Web Text))

Max Activating Token Index: 521

Text #12

Max Range: 2.8456. Min Range: -2.8456

Max Act: 2.1306. Min Act: -0.0000

Data Index: 618550 (C4 (Web Text))

Max Activating Token Index: 254

Text #13

Max Range: 2.8456. Min Range: -2.8456

Max Act: 1.9181. Min Act: -0.0000

Data Index: 905055 (C4 (Web Text))

Max Activating Token Index: 935

Text #14

Max Range: 2.8456. Min Range: -2.8456

Max Act: 2.1349. Min Act: -0.0000

Data Index: 268208 (C4 (Web Text))

Max Activating Token Index: 414

Text #15

Max Range: 2.8456. Min Range: -2.8456

Max Act: 1.9795. Min Act: -0.0000

Data Index: 42503 (C4 (Web Text))

Max Activating Token Index: 665

Text #16

Max Range: 2.8456. Min Range: -2.8456

Max Act: 2.0948. Min Act: -0.0000

Data Index: 908571 (C4 (Web Text))

Max Activating Token Index: 423

Text #17

Max Range: 2.8456. Min Range: -2.8456

Max Act: 2.0933. Min Act: -0.0000

Data Index: 336867 (C4 (Web Text))

Max Activating Token Index: 788

Text #18

Max Range: 2.8456. Min Range: -2.8456

Max Act: 1.9688. Min Act: -0.0000

Data Index: 503180 (C4 (Web Text))

Max Activating Token Index: 665

Text #19

Max Range: 2.8456. Min Range: -2.8456

Max Act: 2.0447. Min Act: -0.0000

Data Index: 588730 (C4 (Web Text))

Max Activating Token Index: 528