Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 337 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 2.6795. Min Range: -2.6795

Max Act: 2.6795. Min Act: -0.0000

Data Index: 484220 (C4 (Web Text))

Max Activating Token Index: 38

Text #1

Max Range: 2.6795. Min Range: -2.6795

Max Act: 2.3703. Min Act: -0.0000

Data Index: 852571 (C4 (Web Text))

Max Activating Token Index: 872

Text #2

Max Range: 2.6795. Min Range: -2.6795

Max Act: 2.5096. Min Act: -0.0000

Data Index: 308957 (C4 (Web Text))

Max Activating Token Index: 312

Text #3

Max Range: 2.6795. Min Range: -2.6795

Max Act: 2.3714. Min Act: -0.0000

Data Index: 1103260 (C4 (Web Text))

Max Activating Token Index: 321

Text #4

Max Range: 2.6795. Min Range: -2.6795

Max Act: 2.3347. Min Act: -0.0000

Data Index: 1050701 (C4 (Web Text))

Max Activating Token Index: 583

Text #5

Max Range: 2.6795. Min Range: -2.6795

Max Act: 2.2966. Min Act: -0.0000

Data Index: 195426 (C4 (Web Text))

Max Activating Token Index: 123

Text #6

Max Range: 2.6795. Min Range: -2.6795

Max Act: 2.2921. Min Act: -0.0000

Data Index: 999589 (C4 (Web Text))

Max Activating Token Index: 212

Text #7

Max Range: 2.6795. Min Range: -2.6795

Max Act: 2.1363. Min Act: -0.0000

Data Index: 495327 (C4 (Web Text))

Max Activating Token Index: 975

Text #8

Max Range: 2.6795. Min Range: -2.6795

Max Act: 2.2027. Min Act: -0.0000

Data Index: 958804 (C4 (Web Text))

Max Activating Token Index: 374

Text #9

Max Range: 2.6795. Min Range: -2.6795

Max Act: 2.1830. Min Act: -0.0000

Data Index: 432390 (C4 (Web Text))

Max Activating Token Index: 889

Text #10

Max Range: 2.6795. Min Range: -2.6795

Max Act: 2.0634. Min Act: -0.0000

Data Index: 899049 (C4 (Web Text))

Max Activating Token Index: 791

Text #11

Max Range: 2.6795. Min Range: -2.6795

Max Act: 2.0617. Min Act: -0.0000

Data Index: 1306109 (C4 (Web Text))

Max Activating Token Index: 30

Text #12

Max Range: 2.6795. Min Range: -2.6795

Max Act: 2.1394. Min Act: -0.0000

Data Index: 1146841 (C4 (Web Text))

Max Activating Token Index: 700

Text #13

Max Range: 2.6795. Min Range: -2.6795

Max Act: 1.9980. Min Act: -0.0000

Data Index: 88544 (C4 (Web Text))

Max Activating Token Index: 231

Text #14

Max Range: 2.6795. Min Range: -2.6795

Max Act: 2.1063. Min Act: -0.0000

Data Index: 492313 (C4 (Web Text))

Max Activating Token Index: 119

Text #15

Max Range: 2.6795. Min Range: -2.6795

Max Act: 2.0761. Min Act: -0.0000

Data Index: 727571 (C4 (Web Text))

Max Activating Token Index: 138

Text #16

Max Range: 2.6795. Min Range: -2.6795

Max Act: 2.0654. Min Act: -0.0000

Data Index: 48107 (C4 (Web Text))

Max Activating Token Index: 441

Text #17

Max Range: 2.6795. Min Range: -2.6795

Max Act: 2.0620. Min Act: -0.0000

Data Index: 18758 (C4 (Web Text))

Max Activating Token Index: 871

Text #18

Max Range: 2.6795. Min Range: -2.6795

Max Act: 2.0462. Min Act: -0.0000

Data Index: 10649 (C4 (Web Text))

Max Activating Token Index: 179

Text #19

Max Range: 2.6795. Min Range: -2.6795

Max Act: 2.0416. Min Act: -0.0000

Data Index: 545573 (C4 (Web Text))

Max Activating Token Index: 243