Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 433 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 5.6765. Min Range: -5.6765

Max Act: 5.6765. Min Act: -0.0000

Data Index: 131230 (C4 (Web Text))

Max Activating Token Index: 110

Text #1

Max Range: 5.6765. Min Range: -5.6765

Max Act: 5.3850. Min Act: -0.0000

Data Index: 152103 (C4 (Web Text))

Max Activating Token Index: 206

Text #2

Max Range: 5.6765. Min Range: -5.6765

Max Act: 5.0414. Min Act: -0.0000

Data Index: 648146 (C4 (Web Text))

Max Activating Token Index: 593

Text #3

Max Range: 5.6765. Min Range: -5.6765

Max Act: 5.3979. Min Act: -0.0000

Data Index: 82514 (C4 (Web Text))

Max Activating Token Index: 810

Text #4

Max Range: 5.6765. Min Range: -5.6765

Max Act: 5.0729. Min Act: -0.0000

Data Index: 314847 (C4 (Web Text))

Max Activating Token Index: 889

Text #5

Max Range: 5.6765. Min Range: -5.6765

Max Act: 5.4266. Min Act: -0.0000

Data Index: 1023827 (C4 (Web Text))

Max Activating Token Index: 179

Text #6

Max Range: 5.6765. Min Range: -5.6765

Max Act: 4.8821. Min Act: -0.0000

Data Index: 799481 (C4 (Web Text))

Max Activating Token Index: 535

Text #7

Max Range: 5.6765. Min Range: -5.6765

Max Act: 5.0436. Min Act: -0.0000

Data Index: 329246 (C4 (Web Text))

Max Activating Token Index: 589

Text #8

Max Range: 5.6765. Min Range: -5.6765

Max Act: 4.4293. Min Act: -0.0000

Data Index: 1123678 (C4 (Web Text))

Max Activating Token Index: 505

Text #9

Max Range: 5.6765. Min Range: -5.6765

Max Act: 4.7681. Min Act: -0.0000

Data Index: 927448 (C4 (Web Text))

Max Activating Token Index: 951

Text #10

Max Range: 5.6765. Min Range: -5.6765

Max Act: 4.6893. Min Act: -0.0000

Data Index: 1044336 (C4 (Web Text))

Max Activating Token Index: 546

Text #11

Max Range: 5.6765. Min Range: -5.6765

Max Act: 4.6403. Min Act: -0.0000

Data Index: 1065676 (C4 (Web Text))

Max Activating Token Index: 206

Text #12

Max Range: 5.6765. Min Range: -5.6765

Max Act: 4.4863. Min Act: -0.0000

Data Index: 945658 (C4 (Web Text))

Max Activating Token Index: 121

Text #13

Max Range: 5.6765. Min Range: -5.6765

Max Act: 4.4064. Min Act: -0.0000

Data Index: 995658 (C4 (Web Text))

Max Activating Token Index: 405

Text #14

Max Range: 5.6765. Min Range: -5.6765

Max Act: 4.2622. Min Act: -0.0000

Data Index: 74140 (C4 (Web Text))

Max Activating Token Index: 331

Text #15

Max Range: 5.6765. Min Range: -5.6765

Max Act: 4.1898. Min Act: -0.0000

Data Index: 634889 (C4 (Web Text))

Max Activating Token Index: 62

Text #16

Max Range: 5.6765. Min Range: -5.6765

Max Act: 4.2344. Min Act: -0.0000

Data Index: 219087 (C4 (Web Text))

Max Activating Token Index: 861

Text #17

Max Range: 5.6765. Min Range: -5.6765

Max Act: 4.3427. Min Act: -0.0000

Data Index: 845338 (C4 (Web Text))

Max Activating Token Index: 484

Text #18

Max Range: 5.6765. Min Range: -5.6765

Max Act: 3.9694. Min Act: -0.0000

Data Index: 1104124 (C4 (Web Text))

Max Activating Token Index: 91

Text #19

Max Range: 5.6765. Min Range: -5.6765

Max Act: 3.8033. Min Act: -0.0000

Data Index: 1300421 (C4 (Web Text))

Max Activating Token Index: 90