Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2830 in Layer 3

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.1906. Min Range: -1.1906

Max Act: 1.1906. Min Act: -0.0001

Data Index: 653498 (C4 (Web Text))

Max Activating Token Index: 689

Text #1

Max Range: 1.1906. Min Range: -1.1906

Max Act: 0.9693. Min Act: -0.0001

Data Index: 301281 (C4 (Web Text))

Max Activating Token Index: 41

Text #2

Max Range: 1.1906. Min Range: -1.1906

Max Act: 0.8805. Min Act: -0.0001

Data Index: 1338539 (C4 (Web Text))

Max Activating Token Index: 855

Text #3

Max Range: 1.1906. Min Range: -1.1906

Max Act: 0.8990. Min Act: -0.0001

Data Index: 1110584 (C4 (Web Text))

Max Activating Token Index: 307

Text #4

Max Range: 1.1906. Min Range: -1.1906

Max Act: 0.8759. Min Act: -0.0001

Data Index: 713282 (C4 (Web Text))

Max Activating Token Index: 719

Text #5

Max Range: 1.1906. Min Range: -1.1906

Max Act: 0.8416. Min Act: -0.0001

Data Index: 91767 (C4 (Web Text))

Max Activating Token Index: 290

Text #6

Max Range: 1.1906. Min Range: -1.1906

Max Act: 0.8830. Min Act: -0.0001

Data Index: 259282 (C4 (Web Text))

Max Activating Token Index: 669

Text #7

Max Range: 1.1906. Min Range: -1.1906

Max Act: 0.8173. Min Act: -0.0001

Data Index: 58055 (C4 (Web Text))

Max Activating Token Index: 834

Text #8

Max Range: 1.1906. Min Range: -1.1906

Max Act: 0.8364. Min Act: -0.0001

Data Index: 224233 (C4 (Web Text))

Max Activating Token Index: 361

Text #9

Max Range: 1.1906. Min Range: -1.1906

Max Act: 0.7518. Min Act: -0.0001

Data Index: 405390 (C4 (Web Text))

Max Activating Token Index: 496

Text #10

Max Range: 1.1906. Min Range: -1.1906

Max Act: 0.6990. Min Act: -0.0001

Data Index: 1321193 (C4 (Web Text))

Max Activating Token Index: 609

Text #11

Max Range: 1.1906. Min Range: -1.1906

Max Act: 0.8531. Min Act: -0.0001

Data Index: 1279376 (C4 (Web Text))

Max Activating Token Index: 38

Text #12

Max Range: 1.1906. Min Range: -1.1906

Max Act: 0.7822. Min Act: -0.0001

Data Index: 79195 (C4 (Web Text))

Max Activating Token Index: 211

Text #13

Max Range: 1.1906. Min Range: -1.1906

Max Act: 0.8045. Min Act: -0.0001

Data Index: 721461 (C4 (Web Text))

Max Activating Token Index: 311

Text #14

Max Range: 1.1906. Min Range: -1.1906

Max Act: 0.7617. Min Act: -0.0001

Data Index: 680138 (C4 (Web Text))

Max Activating Token Index: 95

Text #15

Max Range: 1.1906. Min Range: -1.1906

Max Act: 0.7391. Min Act: -0.0001

Data Index: 1344609 (C4 (Web Text))

Max Activating Token Index: 644

Text #16

Max Range: 1.1906. Min Range: -1.1906

Max Act: 0.8216. Min Act: -0.0001

Data Index: 1056922 (C4 (Web Text))

Max Activating Token Index: 949

Text #17

Max Range: 1.1906. Min Range: -1.1906

Max Act: 0.7217. Min Act: -0.0001

Data Index: 622316 (C4 (Web Text))

Max Activating Token Index: 649

Text #18

Max Range: 1.1906. Min Range: -1.1906

Max Act: 0.7189. Min Act: -0.0001

Data Index: 462685 (C4 (Web Text))

Max Activating Token Index: 912

Text #19

Max Range: 1.1906. Min Range: -1.1906

Max Act: 0.6852. Min Act: -0.0001

Data Index: 776058 (C4 (Web Text))

Max Activating Token Index: 563