Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 5087 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.5110. Min Range: -0.5110

Max Act: 0.5110. Min Act: -0.0000

Data Index: 1202173 (C4 (Web Text))

Max Activating Token Index: 465

Text #1

Max Range: 0.5110. Min Range: -0.5110

Max Act: 0.4849. Min Act: -0.0000

Data Index: 271525 (C4 (Web Text))

Max Activating Token Index: 1022

Text #2

Max Range: 0.5110. Min Range: -0.5110

Max Act: 0.5007. Min Act: -0.0000

Data Index: 770533 (C4 (Web Text))

Max Activating Token Index: 848

Text #3

Max Range: 0.5110. Min Range: -0.5110

Max Act: 0.4601. Min Act: -0.0000

Data Index: 73483 (C4 (Web Text))

Max Activating Token Index: 492

Text #4

Max Range: 0.5110. Min Range: -0.5110

Max Act: 0.4573. Min Act: -0.0000

Data Index: 619091 (C4 (Web Text))

Max Activating Token Index: 427

Text #5

Max Range: 0.5110. Min Range: -0.5110

Max Act: 0.4513. Min Act: -0.0000

Data Index: 916224 (C4 (Web Text))

Max Activating Token Index: 793

Text #6

Max Range: 0.5110. Min Range: -0.5110

Max Act: 0.4509. Min Act: -0.0000

Data Index: 584501 (C4 (Web Text))

Max Activating Token Index: 353

Text #7

Max Range: 0.5110. Min Range: -0.5110

Max Act: 0.4240. Min Act: -0.0000

Data Index: 174899 (C4 (Web Text))

Max Activating Token Index: 948

Text #8

Max Range: 0.5110. Min Range: -0.5110

Max Act: 0.3963. Min Act: -0.0000

Data Index: 1017036 (C4 (Web Text))

Max Activating Token Index: 216

Text #9

Max Range: 0.5110. Min Range: -0.5110

Max Act: 0.4121. Min Act: -0.0000

Data Index: 892501 (C4 (Web Text))

Max Activating Token Index: 1020

Text #10

Max Range: 0.5110. Min Range: -0.5110

Max Act: 0.4276. Min Act: -0.0000

Data Index: 258966 (C4 (Web Text))

Max Activating Token Index: 819

Text #11

Max Range: 0.5110. Min Range: -0.5110

Max Act: 0.4114. Min Act: -0.0000

Data Index: 605839 (C4 (Web Text))

Max Activating Token Index: 265

Text #12

Max Range: 0.5110. Min Range: -0.5110

Max Act: 0.4096. Min Act: -0.0000

Data Index: 1225077 (C4 (Web Text))

Max Activating Token Index: 784

Text #13

Max Range: 0.5110. Min Range: -0.5110

Max Act: 0.3938. Min Act: -0.0000

Data Index: 503207 (C4 (Web Text))

Max Activating Token Index: 392

Text #14

Max Range: 0.5110. Min Range: -0.5110

Max Act: 0.3959. Min Act: -0.0000

Data Index: 712933 (C4 (Web Text))

Max Activating Token Index: 391

Text #15

Max Range: 0.5110. Min Range: -0.5110

Max Act: 0.3940. Min Act: -0.0000

Data Index: 749064 (C4 (Web Text))

Max Activating Token Index: 640

Text #16

Max Range: 0.5110. Min Range: -0.5110

Max Act: 0.3779. Min Act: -0.0000

Data Index: 1208318 (C4 (Web Text))

Max Activating Token Index: 119

Text #17

Max Range: 0.5110. Min Range: -0.5110

Max Act: 0.3806. Min Act: -0.0000

Data Index: 649805 (C4 (Web Text))

Max Activating Token Index: 210

Text #18

Max Range: 0.5110. Min Range: -0.5110

Max Act: 0.3766. Min Act: -0.0000

Data Index: 412311 (C4 (Web Text))

Max Activating Token Index: 703

Text #19

Max Range: 0.5110. Min Range: -0.5110

Max Act: 0.3802. Min Act: -0.0000

Data Index: 653842 (C4 (Web Text))

Max Activating Token Index: 695