Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4231 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.4482. Min Range: -1.4482

Max Act: 1.4436. Min Act: -0.0000

Data Index: 350970 (C4 (Web Text))

Max Activating Token Index: 341

Text #1

Max Range: 1.4482. Min Range: -1.4482

Max Act: 1.3631. Min Act: -0.0000

Data Index: 872454 (C4 (Web Text))

Max Activating Token Index: 361

Text #2

Max Range: 1.4482. Min Range: -1.4482

Max Act: 1.4158. Min Act: -0.0000

Data Index: 437235 (C4 (Web Text))

Max Activating Token Index: 666

Text #3

Max Range: 1.4482. Min Range: -1.4482

Max Act: 1.4190. Min Act: -0.0000

Data Index: 79166 (C4 (Web Text))

Max Activating Token Index: 596

Text #4

Max Range: 1.4482. Min Range: -1.4482

Max Act: 1.4156. Min Act: -0.0000

Data Index: 40536 (C4 (Web Text))

Max Activating Token Index: 417

Text #5

Max Range: 1.4482. Min Range: -1.4482

Max Act: 1.3469. Min Act: -0.0000

Data Index: 136553 (C4 (Web Text))

Max Activating Token Index: 873

Text #6

Max Range: 1.4482. Min Range: -1.4482

Max Act: 1.3319. Min Act: -0.0000

Data Index: 648942 (C4 (Web Text))

Max Activating Token Index: 541

Text #7

Max Range: 1.4482. Min Range: -1.4482

Max Act: 1.4482. Min Act: -0.0000

Data Index: 1142115 (C4 (Web Text))

Max Activating Token Index: 984

Text #8

Max Range: 1.4482. Min Range: -1.4482

Max Act: 1.2950. Min Act: -0.0000

Data Index: 622056 (C4 (Web Text))

Max Activating Token Index: 459

Text #9

Max Range: 1.4482. Min Range: -1.4482

Max Act: 1.2418. Min Act: -0.0000

Data Index: 787562 (C4 (Web Text))

Max Activating Token Index: 249

Text #10

Max Range: 1.4482. Min Range: -1.4482

Max Act: 1.3445. Min Act: -0.0000

Data Index: 1146892 (C4 (Web Text))

Max Activating Token Index: 59

Text #11

Max Range: 1.4482. Min Range: -1.4482

Max Act: 1.2643. Min Act: -0.0000

Data Index: 487401 (C4 (Web Text))

Max Activating Token Index: 936

Text #12

Max Range: 1.4482. Min Range: -1.4482

Max Act: 1.2801. Min Act: -0.0000

Data Index: 1161149 (C4 (Web Text))

Max Activating Token Index: 697

Text #13

Max Range: 1.4482. Min Range: -1.4482

Max Act: 1.3974. Min Act: -0.0000

Data Index: 1140385 (C4 (Web Text))

Max Activating Token Index: 264

Text #14

Max Range: 1.4482. Min Range: -1.4482

Max Act: 1.2458. Min Act: -0.0000

Data Index: 537284 (C4 (Web Text))

Max Activating Token Index: 1006

Text #15

Max Range: 1.4482. Min Range: -1.4482

Max Act: 1.1880. Min Act: -0.0000

Data Index: 259263 (C4 (Web Text))

Max Activating Token Index: 582

Text #16

Max Range: 1.4482. Min Range: -1.4482

Max Act: 1.1330. Min Act: -0.0000

Data Index: 1183657 (C4 (Web Text))

Max Activating Token Index: 700

Text #17

Max Range: 1.4482. Min Range: -1.4482

Max Act: 1.2074. Min Act: -0.0000

Data Index: 1277359 (C4 (Web Text))

Max Activating Token Index: 380

Text #18

Max Range: 1.4482. Min Range: -1.4482

Max Act: 1.2057. Min Act: -0.0000

Data Index: 924213 (C4 (Web Text))

Max Activating Token Index: 372

Text #19

Max Range: 1.4482. Min Range: -1.4482

Max Act: 1.3314. Min Act: -0.0000

Data Index: 1153042 (C4 (Web Text))

Max Activating Token Index: 787