Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 6053 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.9958. Min Range: -1.9958

Max Act: 1.9958. Min Act: -0.0000

Data Index: 1198253 (C4 (Web Text))

Max Activating Token Index: 905

Text #1

Max Range: 1.9958. Min Range: -1.9958

Max Act: 1.9339. Min Act: -0.0000

Data Index: 222450 (C4 (Web Text))

Max Activating Token Index: 526

Text #2

Max Range: 1.9958. Min Range: -1.9958

Max Act: 1.9221. Min Act: -0.0000

Data Index: 48995 (C4 (Web Text))

Max Activating Token Index: 736

Text #3

Max Range: 1.9958. Min Range: -1.9958

Max Act: 1.8393. Min Act: -0.0000

Data Index: 248850 (C4 (Web Text))

Max Activating Token Index: 322

Text #4

Max Range: 1.9958. Min Range: -1.9958

Max Act: 1.8077. Min Act: -0.0000

Data Index: 421725 (C4 (Web Text))

Max Activating Token Index: 122

Text #5

Max Range: 1.9958. Min Range: -1.9958

Max Act: 1.7911. Min Act: -0.0000

Data Index: 425177 (C4 (Web Text))

Max Activating Token Index: 78

Text #6

Max Range: 1.9958. Min Range: -1.9958

Max Act: 1.7850. Min Act: -0.0000

Data Index: 474562 (C4 (Web Text))

Max Activating Token Index: 805

Text #7

Max Range: 1.9958. Min Range: -1.9958

Max Act: 1.7731. Min Act: -0.0000

Data Index: 1346645 (C4 (Web Text))

Max Activating Token Index: 135

Text #8

Max Range: 1.9958. Min Range: -1.9958

Max Act: 1.7019. Min Act: -0.0000

Data Index: 1163566 (C4 (Web Text))

Max Activating Token Index: 679

Text #9

Max Range: 1.9958. Min Range: -1.9958

Max Act: 1.6501. Min Act: -0.0000

Data Index: 315335 (C4 (Web Text))

Max Activating Token Index: 372

Text #10

Max Range: 1.9958. Min Range: -1.9958

Max Act: 1.5942. Min Act: -0.0000

Data Index: 646998 (C4 (Web Text))

Max Activating Token Index: 101

Text #11

Max Range: 1.9958. Min Range: -1.9958

Max Act: 1.6180. Min Act: -0.0000

Data Index: 35594 (C4 (Web Text))

Max Activating Token Index: 142

Text #12

Max Range: 1.9958. Min Range: -1.9958

Max Act: 1.5935. Min Act: -0.0000

Data Index: 748972 (C4 (Web Text))

Max Activating Token Index: 463

Text #13

Max Range: 1.9958. Min Range: -1.9958

Max Act: 1.5714. Min Act: -0.0000

Data Index: 901522 (C4 (Web Text))

Max Activating Token Index: 564

Text #14

Max Range: 1.9958. Min Range: -1.9958

Max Act: 1.7501. Min Act: -0.0000

Data Index: 1322987 (C4 (Web Text))

Max Activating Token Index: 83

Text #15

Max Range: 1.9958. Min Range: -1.9958

Max Act: 1.5574. Min Act: -0.0000

Data Index: 743264 (C4 (Web Text))

Max Activating Token Index: 144

Text #16

Max Range: 1.9958. Min Range: -1.9958

Max Act: 1.4197. Min Act: -0.0000

Data Index: 672719 (C4 (Web Text))

Max Activating Token Index: 364

Text #17

Max Range: 1.9958. Min Range: -1.9958

Max Act: 1.4924. Min Act: -0.0000

Data Index: 22482 (C4 (Web Text))

Max Activating Token Index: 284

Text #18

Max Range: 1.9958. Min Range: -1.9958

Max Act: 1.4680. Min Act: -0.0000

Data Index: 275018 (C4 (Web Text))

Max Activating Token Index: 211

Text #19

Max Range: 1.9958. Min Range: -1.9958

Max Act: 1.3354. Min Act: -0.0000

Data Index: 531298 (C4 (Web Text))

Max Activating Token Index: 79