Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4929 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 2.9542. Min Range: -2.9542

Max Act: 2.9298. Min Act: -0.0000

Data Index: 1307054 (C4 (Web Text))

Max Activating Token Index: 34

Text #1

Max Range: 2.9542. Min Range: -2.9542

Max Act: 2.7507. Min Act: -0.0000

Data Index: 645823 (C4 (Web Text))

Max Activating Token Index: 552

Text #2

Max Range: 2.9542. Min Range: -2.9542

Max Act: 2.9542. Min Act: -0.0000

Data Index: 1350331 (C4 (Web Text))

Max Activating Token Index: 845

Text #3

Max Range: 2.9542. Min Range: -2.9542

Max Act: 2.6424. Min Act: -0.0000

Data Index: 90226 (C4 (Web Text))

Max Activating Token Index: 897

Text #4

Max Range: 2.9542. Min Range: -2.9542

Max Act: 2.6010. Min Act: -0.0000

Data Index: 658177 (C4 (Web Text))

Max Activating Token Index: 736

Text #5

Max Range: 2.9542. Min Range: -2.9542

Max Act: 2.6872. Min Act: -0.0000

Data Index: 1155540 (C4 (Web Text))

Max Activating Token Index: 954

Text #6

Max Range: 2.9542. Min Range: -2.9542

Max Act: 2.6123. Min Act: -0.0000

Data Index: 167388 (C4 (Web Text))

Max Activating Token Index: 707

Text #7

Max Range: 2.9542. Min Range: -2.9542

Max Act: 2.4926. Min Act: -0.0000

Data Index: 121348 (C4 (Web Text))

Max Activating Token Index: 628

Text #8

Max Range: 2.9542. Min Range: -2.9542

Max Act: 2.4419. Min Act: -0.0000

Data Index: 1132356 (C4 (Web Text))

Max Activating Token Index: 751

Text #9

Max Range: 2.9542. Min Range: -2.9542

Max Act: 2.4500. Min Act: -0.0000

Data Index: 361110 (C4 (Web Text))

Max Activating Token Index: 205

Text #10

Max Range: 2.9542. Min Range: -2.9542

Max Act: 2.4186. Min Act: -0.0000

Data Index: 66471 (C4 (Web Text))

Max Activating Token Index: 525

Text #11

Max Range: 2.9542. Min Range: -2.9542

Max Act: 2.2876. Min Act: -0.0000

Data Index: 539667 (C4 (Web Text))

Max Activating Token Index: 747

Text #12

Max Range: 2.9542. Min Range: -2.9542

Max Act: 2.1774. Min Act: -0.0000

Data Index: 182868 (C4 (Web Text))

Max Activating Token Index: 779

Text #13

Max Range: 2.9542. Min Range: -2.9542

Max Act: 2.5130. Min Act: -0.0000

Data Index: 320261 (C4 (Web Text))

Max Activating Token Index: 221

Text #14

Max Range: 2.9542. Min Range: -2.9542

Max Act: 2.3547. Min Act: -0.0000

Data Index: 404809 (C4 (Web Text))

Max Activating Token Index: 714

Text #15

Max Range: 2.9542. Min Range: -2.9542

Max Act: 2.3371. Min Act: -0.0000

Data Index: 104281 (C4 (Web Text))

Max Activating Token Index: 915

Text #16

Max Range: 2.9542. Min Range: -2.9542

Max Act: 2.4253. Min Act: -0.0000

Data Index: 1067027 (C4 (Web Text))

Max Activating Token Index: 201

Text #17

Max Range: 2.9542. Min Range: -2.9542

Max Act: 2.1689. Min Act: -0.0000

Data Index: 58805 (C4 (Web Text))

Max Activating Token Index: 432

Text #18

Max Range: 2.9542. Min Range: -2.9542

Max Act: 2.2950. Min Act: -0.0000

Data Index: 270471 (C4 (Web Text))

Max Activating Token Index: 761

Text #19

Max Range: 2.9542. Min Range: -2.9542

Max Act: 2.2702. Min Act: -0.0000

Data Index: 650196 (C4 (Web Text))

Max Activating Token Index: 368