Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 5867 in Layer 4

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 3.0086. Min Range: -3.0086

Max Act: 2.8190. Min Act: -0.0000

Data Index: 841762 (C4 (Web Text))

Max Activating Token Index: 696

Text #1

Max Range: 3.0086. Min Range: -3.0086

Max Act: 2.8885. Min Act: -0.0000

Data Index: 333637 (C4 (Web Text))

Max Activating Token Index: 676

Text #2

Max Range: 3.0086. Min Range: -3.0086

Max Act: 3.0086. Min Act: -0.0000

Data Index: 296308 (C4 (Web Text))

Max Activating Token Index: 195

Text #3

Max Range: 3.0086. Min Range: -3.0086

Max Act: 2.5625. Min Act: -0.0000

Data Index: 846438 (C4 (Web Text))

Max Activating Token Index: 650

Text #4

Max Range: 3.0086. Min Range: -3.0086

Max Act: 2.6198. Min Act: -0.0000

Data Index: 469387 (C4 (Web Text))

Max Activating Token Index: 909

Text #5

Max Range: 3.0086. Min Range: -3.0086

Max Act: 2.6057. Min Act: -0.0000

Data Index: 69606 (C4 (Web Text))

Max Activating Token Index: 403

Text #6

Max Range: 3.0086. Min Range: -3.0086

Max Act: 2.2218. Min Act: -0.0000

Data Index: 1196982 (C4 (Web Text))

Max Activating Token Index: 51

Text #7

Max Range: 3.0086. Min Range: -3.0086

Max Act: 2.3279. Min Act: -0.0000

Data Index: 23879 (C4 (Web Text))

Max Activating Token Index: 969

Text #8

Max Range: 3.0086. Min Range: -3.0086

Max Act: 2.2610. Min Act: -0.0000

Data Index: 623531 (C4 (Web Text))

Max Activating Token Index: 336

Text #9

Max Range: 3.0086. Min Range: -3.0086

Max Act: 2.3971. Min Act: -0.0000

Data Index: 678659 (C4 (Web Text))

Max Activating Token Index: 776

Text #10

Max Range: 3.0086. Min Range: -3.0086

Max Act: 2.2749. Min Act: -0.0001

Data Index: 530739 (C4 (Web Text))

Max Activating Token Index: 496

Text #11

Max Range: 3.0086. Min Range: -3.0086

Max Act: 2.3123. Min Act: -0.0000

Data Index: 108340 (C4 (Web Text))

Max Activating Token Index: 260

Text #12

Max Range: 3.0086. Min Range: -3.0086

Max Act: 2.1956. Min Act: -0.0000

Data Index: 659425 (C4 (Web Text))

Max Activating Token Index: 137

Text #13

Max Range: 3.0086. Min Range: -3.0086

Max Act: 2.2954. Min Act: -0.0001

Data Index: 806316 (C4 (Web Text))

Max Activating Token Index: 937

Text #14

Max Range: 3.0086. Min Range: -3.0086

Max Act: 2.1257. Min Act: -0.0000

Data Index: 1230192 (C4 (Web Text))

Max Activating Token Index: 174

Text #15

Max Range: 3.0086. Min Range: -3.0086

Max Act: 2.2325. Min Act: -0.0001

Data Index: 29409 (C4 (Web Text))

Max Activating Token Index: 185

Text #16

Max Range: 3.0086. Min Range: -3.0086

Max Act: 2.2172. Min Act: -0.0000

Data Index: 1106876 (C4 (Web Text))

Max Activating Token Index: 19

Text #17

Max Range: 3.0086. Min Range: -3.0086

Max Act: 2.1004. Min Act: -0.0001

Data Index: 1150286 (C4 (Web Text))

Max Activating Token Index: 177

Text #18

Max Range: 3.0086. Min Range: -3.0086

Max Act: 2.1003. Min Act: -0.0000

Data Index: 1308240 (C4 (Web Text))

Max Activating Token Index: 164

Text #19

Max Range: 3.0086. Min Range: -3.0086

Max Act: 2.0666. Min Act: -0.0000

Data Index: 850944 (C4 (Web Text))

Max Activating Token Index: 958