Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 505 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 4.3233. Min Range: -4.3233

Max Act: 4.2501. Min Act: -0.0000

Data Index: 1086157 (C4 (Web Text))

Max Activating Token Index: 703

Text #1

Max Range: 4.3233. Min Range: -4.3233

Max Act: 4.3233. Min Act: -0.0000

Data Index: 419759 (C4 (Web Text))

Max Activating Token Index: 24

Text #2

Max Range: 4.3233. Min Range: -4.3233

Max Act: 3.9499. Min Act: -0.0000

Data Index: 376880 (C4 (Web Text))

Max Activating Token Index: 601

Text #3

Max Range: 4.3233. Min Range: -4.3233

Max Act: 3.5661. Min Act: -0.0000

Data Index: 961710 (C4 (Web Text))

Max Activating Token Index: 886

Text #4

Max Range: 4.3233. Min Range: -4.3233

Max Act: 3.6742. Min Act: -0.0000

Data Index: 939189 (C4 (Web Text))

Max Activating Token Index: 19

Text #5

Max Range: 4.3233. Min Range: -4.3233

Max Act: 3.8226. Min Act: -0.0000

Data Index: 637606 (C4 (Web Text))

Max Activating Token Index: 49

Text #6

Max Range: 4.3233. Min Range: -4.3233

Max Act: 3.4997. Min Act: -0.0000

Data Index: 126137 (C4 (Web Text))

Max Activating Token Index: 19

Text #7

Max Range: 4.3233. Min Range: -4.3233

Max Act: 3.2749. Min Act: -0.0000

Data Index: 73776 (C4 (Web Text))

Max Activating Token Index: 173

Text #8

Max Range: 4.3233. Min Range: -4.3233

Max Act: 3.3904. Min Act: -0.0000

Data Index: 532320 (C4 (Web Text))

Max Activating Token Index: 249

Text #9

Max Range: 4.3233. Min Range: -4.3233

Max Act: 3.4949. Min Act: -0.0000

Data Index: 404434 (C4 (Web Text))

Max Activating Token Index: 139

Text #10

Max Range: 4.3233. Min Range: -4.3233

Max Act: 3.7128. Min Act: -0.0000

Data Index: 989136 (C4 (Web Text))

Max Activating Token Index: 122

Text #11

Max Range: 4.3233. Min Range: -4.3233

Max Act: 3.3087. Min Act: -0.0000

Data Index: 261760 (C4 (Web Text))

Max Activating Token Index: 541

Text #12

Max Range: 4.3233. Min Range: -4.3233

Max Act: 3.2893. Min Act: -0.0000

Data Index: 518069 (C4 (Web Text))

Max Activating Token Index: 543

Text #13

Max Range: 4.3233. Min Range: -4.3233

Max Act: 3.7641. Min Act: -0.0000

Data Index: 118315 (C4 (Web Text))

Max Activating Token Index: 137

Text #14

Max Range: 4.3233. Min Range: -4.3233

Max Act: 3.1832. Min Act: -0.0000

Data Index: 519181 (C4 (Web Text))

Max Activating Token Index: 229

Text #15

Max Range: 4.3233. Min Range: -4.3233

Max Act: 3.0407. Min Act: -0.0000

Data Index: 1104598 (C4 (Web Text))

Max Activating Token Index: 99

Text #16

Max Range: 4.3233. Min Range: -4.3233

Max Act: 3.1899. Min Act: -0.0000

Data Index: 1112271 (C4 (Web Text))

Max Activating Token Index: 53

Text #17

Max Range: 4.3233. Min Range: -4.3233

Max Act: 3.1558. Min Act: -0.0000

Data Index: 1357410 (C4 (Web Text))

Max Activating Token Index: 335

Text #18

Max Range: 4.3233. Min Range: -4.3233

Max Act: 3.0759. Min Act: -0.0000

Data Index: 1145710 (C4 (Web Text))

Max Activating Token Index: 328

Text #19

Max Range: 4.3233. Min Range: -4.3233

Max Act: 3.1783. Min Act: -0.0000

Data Index: 170739 (C4 (Web Text))

Max Activating Token Index: 581