Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 5570 in Layer 1

Load this data into an Interactive Neuroscope

See Documentation here

Hooked Transformer Loading: HookedTransformer.from_pretrained('solu-12l')

Click the toggle to see the full text!

Text #0

Max Range: 1.6315. Min Range: -1.6315

Max Act: 1.6315. Min Act: -0.0002

Data Index: 859470 (C4 (Web Text))

Max Activating Token Index: 41

Text #1

Max Range: 1.6315. Min Range: -1.6315

Max Act: 1.5607. Min Act: -0.0002

Data Index: 1079218 (C4 (Web Text))

Max Activating Token Index: 678

Text #2

Max Range: 1.6315. Min Range: -1.6315

Max Act: 1.5011. Min Act: -0.0002

Data Index: 847585 (C4 (Web Text))

Max Activating Token Index: 858

Text #3

Max Range: 1.6315. Min Range: -1.6315

Max Act: 1.4871. Min Act: -0.0002

Data Index: 236117 (C4 (Web Text))

Max Activating Token Index: 122

Text #4

Max Range: 1.6315. Min Range: -1.6315

Max Act: 1.5747. Min Act: -0.0002

Data Index: 316479 (C4 (Web Text))

Max Activating Token Index: 1020

Text #5

Max Range: 1.6315. Min Range: -1.6315

Max Act: 1.4917. Min Act: -0.0002

Data Index: 107980 (C4 (Web Text))

Max Activating Token Index: 449

Text #6

Max Range: 1.6315. Min Range: -1.6315

Max Act: 1.4382. Min Act: -0.0002

Data Index: 648013 (C4 (Web Text))

Max Activating Token Index: 354

Text #7

Max Range: 1.6315. Min Range: -1.6315

Max Act: 1.4842. Min Act: -0.0002

Data Index: 1178752 (C4 (Web Text))

Max Activating Token Index: 748

Text #8

Max Range: 1.6315. Min Range: -1.6315

Max Act: 1.4733. Min Act: -0.0002

Data Index: 1047159 (C4 (Web Text))

Max Activating Token Index: 816

Text #9

Max Range: 1.6315. Min Range: -1.6315

Max Act: 1.4376. Min Act: -0.0002

Data Index: 1108512 (C4 (Web Text))

Max Activating Token Index: 485

Text #10

Max Range: 1.6315. Min Range: -1.6315

Max Act: 1.4276. Min Act: -0.0002

Data Index: 175596 (C4 (Web Text))

Max Activating Token Index: 898

Text #11

Max Range: 1.6315. Min Range: -1.6315

Max Act: 1.4445. Min Act: -0.0002

Data Index: 978903 (C4 (Web Text))

Max Activating Token Index: 644

Text #12

Max Range: 1.6315. Min Range: -1.6315

Max Act: 1.4439. Min Act: -0.0002

Data Index: 1037674 (C4 (Web Text))

Max Activating Token Index: 927

Text #13

Max Range: 1.6315. Min Range: -1.6315

Max Act: 1.4917. Min Act: -0.0002

Data Index: 1255070 (C4 (Web Text))

Max Activating Token Index: 897

Text #14

Max Range: 1.6315. Min Range: -1.6315

Max Act: 1.4524. Min Act: -0.0002

Data Index: 1218190 (C4 (Web Text))

Max Activating Token Index: 756

Text #15

Max Range: 1.6315. Min Range: -1.6315

Max Act: 1.4262. Min Act: -0.0002

Data Index: 85708 (C4 (Web Text))

Max Activating Token Index: 155

Text #16

Max Range: 1.6315. Min Range: -1.6315

Max Act: 1.4271. Min Act: -0.0002

Data Index: 953367 (C4 (Web Text))

Max Activating Token Index: 70

Text #17

Max Range: 1.6315. Min Range: -1.6315

Max Act: 1.3927. Min Act: -0.0002

Data Index: 1034408 (C4 (Web Text))

Max Activating Token Index: 49

Text #18

Max Range: 1.6315. Min Range: -1.6315

Max Act: 1.4252. Min Act: -0.0002

Data Index: 120926 (C4 (Web Text))

Max Activating Token Index: 274

Text #19

Max Range: 1.6315. Min Range: -1.6315

Max Act: 1.3780. Min Act: -0.0002

Data Index: 1025243 (C4 (Web Text))

Max Activating Token Index: 912