Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4558 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 2.3924. Min Range: -2.3924

Max Act: 2.3772. Min Act: -0.0000

Data Index: 1615834 (Python Code)

Max Activating Token Index: 851

Text #1

Max Range: 2.3924. Min Range: -2.3924

Max Act: 2.3924. Min Act: -0.0000

Data Index: 310923 (C4 (Web Text))

Max Activating Token Index: 911

Text #2

Max Range: 2.3924. Min Range: -2.3924

Max Act: 2.1809. Min Act: -0.0000

Data Index: 1442176 (Python Code)

Max Activating Token Index: 162

Text #3

Max Range: 2.3924. Min Range: -2.3924

Max Act: 2.1061. Min Act: -0.0000

Data Index: 760735 (C4 (Web Text))

Max Activating Token Index: 941

Text #4

Max Range: 2.3924. Min Range: -2.3924

Max Act: 2.0709. Min Act: -0.0000

Data Index: 1069654 (C4 (Web Text))

Max Activating Token Index: 421

Text #5

Max Range: 2.3924. Min Range: -2.3924

Max Act: 2.0244. Min Act: -0.0000

Data Index: 286127 (C4 (Web Text))

Max Activating Token Index: 516

Text #6

Max Range: 2.3924. Min Range: -2.3924

Max Act: 2.0152. Min Act: -0.0000

Data Index: 75771 (C4 (Web Text))

Max Activating Token Index: 234

Text #7

Max Range: 2.3924. Min Range: -2.3924

Max Act: 2.0596. Min Act: -0.0000

Data Index: 965582 (C4 (Web Text))

Max Activating Token Index: 73

Text #8

Max Range: 2.3924. Min Range: -2.3924

Max Act: 2.0921. Min Act: -0.0000

Data Index: 1091536 (C4 (Web Text))

Max Activating Token Index: 943

Text #9

Max Range: 2.3924. Min Range: -2.3924

Max Act: 1.9083. Min Act: -0.0000

Data Index: 418140 (C4 (Web Text))

Max Activating Token Index: 586

Text #10

Max Range: 2.3924. Min Range: -2.3924

Max Act: 1.8913. Min Act: -0.0000

Data Index: 1235797 (C4 (Web Text))

Max Activating Token Index: 358

Text #11

Max Range: 2.3924. Min Range: -2.3924

Max Act: 1.9980. Min Act: -0.0000

Data Index: 57259 (C4 (Web Text))

Max Activating Token Index: 921

Text #12

Max Range: 2.3924. Min Range: -2.3924

Max Act: 1.9994. Min Act: -0.0000

Data Index: 233697 (C4 (Web Text))

Max Activating Token Index: 338

Text #13

Max Range: 2.3924. Min Range: -2.3924

Max Act: 1.9322. Min Act: -0.0000

Data Index: 861991 (C4 (Web Text))

Max Activating Token Index: 1008

Text #14

Max Range: 2.3924. Min Range: -2.3924

Max Act: 1.9737. Min Act: -0.0000

Data Index: 662537 (C4 (Web Text))

Max Activating Token Index: 77

Text #15

Max Range: 2.3924. Min Range: -2.3924

Max Act: 1.9056. Min Act: -0.0000

Data Index: 11280 (C4 (Web Text))

Max Activating Token Index: 584

Text #16

Max Range: 2.3924. Min Range: -2.3924

Max Act: 1.8641. Min Act: -0.0000

Data Index: 566279 (C4 (Web Text))

Max Activating Token Index: 160

Text #17

Max Range: 2.3924. Min Range: -2.3924

Max Act: 1.9457. Min Act: -0.0000

Data Index: 520455 (C4 (Web Text))

Max Activating Token Index: 886

Text #18

Max Range: 2.3924. Min Range: -2.3924

Max Act: 1.8795. Min Act: -0.0000

Data Index: 673071 (C4 (Web Text))

Max Activating Token Index: 321

Text #19

Max Range: 2.3924. Min Range: -2.3924

Max Act: 1.8133. Min Act: -0.0000

Data Index: 135581 (C4 (Web Text))

Max Activating Token Index: 404