Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3485 in Layer 1

Load this data into an Interactive Neuroscope

See Documentation here

Hooked Transformer Loading: HookedTransformer.from_pretrained('solu-12l')

Click the toggle to see the full text!

Text #0

Max Range: 1.6459. Min Range: -1.6459

Max Act: 1.6459. Min Act: -0.0002

Data Index: 330608 (C4 (Web Text))

Max Activating Token Index: 302

Text #1

Max Range: 1.6459. Min Range: -1.6459

Max Act: 1.6070. Min Act: -0.0002

Data Index: 171876 (C4 (Web Text))

Max Activating Token Index: 520

Text #2

Max Range: 1.6459. Min Range: -1.6459

Max Act: 1.6357. Min Act: -0.0002

Data Index: 1158175 (C4 (Web Text))

Max Activating Token Index: 337

Text #3

Max Range: 1.6459. Min Range: -1.6459

Max Act: 1.6121. Min Act: -0.0002

Data Index: 877930 (C4 (Web Text))

Max Activating Token Index: 320

Text #4

Max Range: 1.6459. Min Range: -1.6459

Max Act: 1.5950. Min Act: -0.0002

Data Index: 334247 (C4 (Web Text))

Max Activating Token Index: 870

Text #5

Max Range: 1.6459. Min Range: -1.6459

Max Act: 1.5815. Min Act: -0.0002

Data Index: 1261910 (C4 (Web Text))

Max Activating Token Index: 321

Text #6

Max Range: 1.6459. Min Range: -1.6459

Max Act: 1.5874. Min Act: -0.0002

Data Index: 364937 (C4 (Web Text))

Max Activating Token Index: 532

Text #7

Max Range: 1.6459. Min Range: -1.6459

Max Act: 1.6265. Min Act: -0.0002

Data Index: 1172830 (C4 (Web Text))

Max Activating Token Index: 136

Text #8

Max Range: 1.6459. Min Range: -1.6459

Max Act: 1.5618. Min Act: -0.0002

Data Index: 584853 (C4 (Web Text))

Max Activating Token Index: 8

Text #9

Max Range: 1.6459. Min Range: -1.6459

Max Act: 1.5714. Min Act: -0.0002

Data Index: 1212637 (C4 (Web Text))

Max Activating Token Index: 462

Text #10

Max Range: 1.6459. Min Range: -1.6459

Max Act: 1.5599. Min Act: -0.0002

Data Index: 373618 (C4 (Web Text))

Max Activating Token Index: 831

Text #11

Max Range: 1.6459. Min Range: -1.6459

Max Act: 1.5374. Min Act: -0.0002

Data Index: 403382 (C4 (Web Text))

Max Activating Token Index: 13

Text #12

Max Range: 1.6459. Min Range: -1.6459

Max Act: 1.5829. Min Act: -0.0002

Data Index: 1057316 (C4 (Web Text))

Max Activating Token Index: 294

Text #13

Max Range: 1.6459. Min Range: -1.6459

Max Act: 1.5588. Min Act: -0.0002

Data Index: 1150290 (C4 (Web Text))

Max Activating Token Index: 752

Text #14

Max Range: 1.6459. Min Range: -1.6459

Max Act: 1.5519. Min Act: -0.0002

Data Index: 295566 (C4 (Web Text))

Max Activating Token Index: 477

Text #15

Max Range: 1.6459. Min Range: -1.6459

Max Act: 1.5226. Min Act: -0.0002

Data Index: 80787 (C4 (Web Text))

Max Activating Token Index: 346

Text #16

Max Range: 1.6459. Min Range: -1.6459

Max Act: 1.5262. Min Act: -0.0002

Data Index: 1333271 (C4 (Web Text))

Max Activating Token Index: 329

Text #17

Max Range: 1.6459. Min Range: -1.6459

Max Act: 1.5091. Min Act: -0.0002

Data Index: 87909 (C4 (Web Text))

Max Activating Token Index: 192

Text #18

Max Range: 1.6459. Min Range: -1.6459

Max Act: 1.4756. Min Act: -0.0002

Data Index: 1297248 (C4 (Web Text))

Max Activating Token Index: 5

Text #19

Max Range: 1.6459. Min Range: -1.6459

Max Act: 1.4999. Min Act: -0.0002

Data Index: 160832 (C4 (Web Text))

Max Activating Token Index: 30