Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 407 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.5121. Min Range: -0.5121

Max Act: 0.5121. Min Act: -0.0000

Data Index: 632806 (C4 (Web Text))

Max Activating Token Index: 563

Text #1

Max Range: 0.5121. Min Range: -0.5121

Max Act: 0.4707. Min Act: -0.0000

Data Index: 1194166 (C4 (Web Text))

Max Activating Token Index: 356

Text #2

Max Range: 0.5121. Min Range: -0.5121

Max Act: 0.4643. Min Act: -0.0000

Data Index: 164266 (C4 (Web Text))

Max Activating Token Index: 129

Text #3

Max Range: 0.5121. Min Range: -0.5121

Max Act: 0.4491. Min Act: -0.0000

Data Index: 160301 (C4 (Web Text))

Max Activating Token Index: 337

Text #4

Max Range: 0.5121. Min Range: -0.5121

Max Act: 0.4061. Min Act: -0.0000

Data Index: 493337 (C4 (Web Text))

Max Activating Token Index: 702

Text #5

Max Range: 0.5121. Min Range: -0.5121

Max Act: 0.3816. Min Act: -0.0000

Data Index: 1029246 (C4 (Web Text))

Max Activating Token Index: 326

Text #6

Max Range: 0.5121. Min Range: -0.5121

Max Act: 0.3752. Min Act: -0.0000

Data Index: 1166300 (C4 (Web Text))

Max Activating Token Index: 501

Text #7

Max Range: 0.5121. Min Range: -0.5121

Max Act: 0.3734. Min Act: -0.0000

Data Index: 102998 (C4 (Web Text))

Max Activating Token Index: 742

Text #8

Max Range: 0.5121. Min Range: -0.5121

Max Act: 0.3735. Min Act: -0.0000

Data Index: 370711 (C4 (Web Text))

Max Activating Token Index: 328

Text #9

Max Range: 0.5121. Min Range: -0.5121

Max Act: 0.3314. Min Act: -0.0000

Data Index: 606348 (C4 (Web Text))

Max Activating Token Index: 113

Text #10

Max Range: 0.5121. Min Range: -0.5121

Max Act: 0.3497. Min Act: -0.0000

Data Index: 494122 (C4 (Web Text))

Max Activating Token Index: 688

Text #11

Max Range: 0.5121. Min Range: -0.5121

Max Act: 0.3404. Min Act: -0.0000

Data Index: 967322 (C4 (Web Text))

Max Activating Token Index: 417

Text #12

Max Range: 0.5121. Min Range: -0.5121

Max Act: 0.3255. Min Act: 0.0000

Data Index: 677171 (C4 (Web Text))

Max Activating Token Index: 318

Text #13

Max Range: 0.5121. Min Range: -0.5121

Max Act: 0.3090. Min Act: -0.0000

Data Index: 838495 (C4 (Web Text))

Max Activating Token Index: 658

Text #14

Max Range: 0.5121. Min Range: -0.5121

Max Act: 0.3203. Min Act: -0.0000

Data Index: 107254 (C4 (Web Text))

Max Activating Token Index: 300

Text #15

Max Range: 0.5121. Min Range: -0.5121

Max Act: 0.3197. Min Act: -0.0000

Data Index: 896787 (C4 (Web Text))

Max Activating Token Index: 516

Text #16

Max Range: 0.5121. Min Range: -0.5121

Max Act: 0.2938. Min Act: -0.0000

Data Index: 315870 (C4 (Web Text))

Max Activating Token Index: 335

Text #17

Max Range: 0.5121. Min Range: -0.5121

Max Act: 0.3003. Min Act: -0.0000

Data Index: 1244709 (C4 (Web Text))

Max Activating Token Index: 501

Text #18

Max Range: 0.5121. Min Range: -0.5121

Max Act: 0.3189. Min Act: -0.0000

Data Index: 576 (C4 (Web Text))

Max Activating Token Index: 154

Text #19

Max Range: 0.5121. Min Range: -0.5121

Max Act: 0.3011. Min Act: -0.0000

Data Index: 293221 (C4 (Web Text))

Max Activating Token Index: 332