Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1284 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 4.2165. Min Range: -4.2165

Max Act: 4.2165. Min Act: -0.0000

Data Index: 1146780 (C4 (Web Text))

Max Activating Token Index: 982

Text #1

Max Range: 4.2165. Min Range: -4.2165

Max Act: 3.8217. Min Act: -0.0000

Data Index: 627200 (C4 (Web Text))

Max Activating Token Index: 430

Text #2

Max Range: 4.2165. Min Range: -4.2165

Max Act: 3.7234. Min Act: -0.0000

Data Index: 1119895 (C4 (Web Text))

Max Activating Token Index: 301

Text #3

Max Range: 4.2165. Min Range: -4.2165

Max Act: 3.6839. Min Act: -0.0000

Data Index: 1036631 (C4 (Web Text))

Max Activating Token Index: 942

Text #4

Max Range: 4.2165. Min Range: -4.2165

Max Act: 3.8328. Min Act: -0.0000

Data Index: 1282525 (C4 (Web Text))

Max Activating Token Index: 770

Text #5

Max Range: 4.2165. Min Range: -4.2165

Max Act: 3.5439. Min Act: -0.0000

Data Index: 886665 (C4 (Web Text))

Max Activating Token Index: 233

Text #6

Max Range: 4.2165. Min Range: -4.2165

Max Act: 3.6149. Min Act: -0.0000

Data Index: 1310444 (C4 (Web Text))

Max Activating Token Index: 205

Text #7

Max Range: 4.2165. Min Range: -4.2165

Max Act: 3.4234. Min Act: -0.0000

Data Index: 9186 (C4 (Web Text))

Max Activating Token Index: 548

Text #8

Max Range: 4.2165. Min Range: -4.2165

Max Act: 3.2758. Min Act: -0.0000

Data Index: 3096 (C4 (Web Text))

Max Activating Token Index: 1013

Text #9

Max Range: 4.2165. Min Range: -4.2165

Max Act: 3.1130. Min Act: -0.0000

Data Index: 1210088 (C4 (Web Text))

Max Activating Token Index: 816

Text #10

Max Range: 4.2165. Min Range: -4.2165

Max Act: 3.1543. Min Act: -0.0000

Data Index: 339026 (C4 (Web Text))

Max Activating Token Index: 637

Text #11

Max Range: 4.2165. Min Range: -4.2165

Max Act: 2.9766. Min Act: -0.0000

Data Index: 1359520 (C4 (Web Text))

Max Activating Token Index: 767

Text #12

Max Range: 4.2165. Min Range: -4.2165

Max Act: 3.2540. Min Act: -0.0000

Data Index: 933559 (C4 (Web Text))

Max Activating Token Index: 1021

Text #13

Max Range: 4.2165. Min Range: -4.2165

Max Act: 2.8795. Min Act: -0.0000

Data Index: 67790 (C4 (Web Text))

Max Activating Token Index: 823

Text #14

Max Range: 4.2165. Min Range: -4.2165

Max Act: 2.9024. Min Act: -0.0000

Data Index: 346656 (C4 (Web Text))

Max Activating Token Index: 592

Text #15

Max Range: 4.2165. Min Range: -4.2165

Max Act: 2.7384. Min Act: -0.0000

Data Index: 316803 (C4 (Web Text))

Max Activating Token Index: 745

Text #16

Max Range: 4.2165. Min Range: -4.2165

Max Act: 2.8307. Min Act: -0.0000

Data Index: 29686 (C4 (Web Text))

Max Activating Token Index: 400

Text #17

Max Range: 4.2165. Min Range: -4.2165

Max Act: 2.8332. Min Act: -0.0000

Data Index: 66297 (C4 (Web Text))

Max Activating Token Index: 930

Text #18

Max Range: 4.2165. Min Range: -4.2165

Max Act: 2.7998. Min Act: -0.0000

Data Index: 278334 (C4 (Web Text))

Max Activating Token Index: 895

Text #19

Max Range: 4.2165. Min Range: -4.2165

Max Act: 2.5753. Min Act: -0.0000

Data Index: 1218316 (C4 (Web Text))

Max Activating Token Index: 885