Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3902 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 0.0692. Min Range: -0.0692

Max Act: 0.0692. Min Act: -0.0002

Data Index: 978154 (C4 (Web Text))

Max Activating Token Index: 140

Text #1

Max Range: 0.0692. Min Range: -0.0692

Max Act: 0.0663. Min Act: -0.0002

Data Index: 1115167 (C4 (Web Text))

Max Activating Token Index: 807

Text #2

Max Range: 0.0692. Min Range: -0.0692

Max Act: 0.0584. Min Act: -0.0002

Data Index: 1296566 (C4 (Web Text))

Max Activating Token Index: 825

Text #3

Max Range: 0.0692. Min Range: -0.0692

Max Act: 0.0559. Min Act: -0.0002

Data Index: 270705 (C4 (Web Text))

Max Activating Token Index: 75

Text #4

Max Range: 0.0692. Min Range: -0.0692

Max Act: 0.0544. Min Act: -0.0002

Data Index: 938559 (C4 (Web Text))

Max Activating Token Index: 796

Text #5

Max Range: 0.0692. Min Range: -0.0692

Max Act: 0.0549. Min Act: -0.0002

Data Index: 258511 (C4 (Web Text))

Max Activating Token Index: 300

Text #6

Max Range: 0.0692. Min Range: -0.0692

Max Act: 0.0512. Min Act: -0.0002

Data Index: 1039617 (C4 (Web Text))

Max Activating Token Index: 350

Text #7

Max Range: 0.0692. Min Range: -0.0692

Max Act: 0.0481. Min Act: -0.0002

Data Index: 1160371 (C4 (Web Text))

Max Activating Token Index: 529

Text #8

Max Range: 0.0692. Min Range: -0.0692

Max Act: 0.0472. Min Act: -0.0002

Data Index: 554023 (C4 (Web Text))

Max Activating Token Index: 320

Text #9

Max Range: 0.0692. Min Range: -0.0692

Max Act: 0.0493. Min Act: -0.0002

Data Index: 102165 (C4 (Web Text))

Max Activating Token Index: 631

Text #10

Max Range: 0.0692. Min Range: -0.0692

Max Act: 0.0490. Min Act: -0.0002

Data Index: 3509 (C4 (Web Text))

Max Activating Token Index: 60

Text #11

Max Range: 0.0692. Min Range: -0.0692

Max Act: 0.0481. Min Act: -0.0002

Data Index: 60134 (C4 (Web Text))

Max Activating Token Index: 247

Text #12

Max Range: 0.0692. Min Range: -0.0692

Max Act: 0.0472. Min Act: -0.0002

Data Index: 966165 (C4 (Web Text))

Max Activating Token Index: 514

Text #13

Max Range: 0.0692. Min Range: -0.0692

Max Act: 0.0462. Min Act: -0.0002

Data Index: 1092621 (C4 (Web Text))

Max Activating Token Index: 187

Text #14

Max Range: 0.0692. Min Range: -0.0692

Max Act: 0.0458. Min Act: -0.0002

Data Index: 995335 (C4 (Web Text))

Max Activating Token Index: 512

Text #15

Max Range: 0.0692. Min Range: -0.0692

Max Act: 0.0438. Min Act: -0.0002

Data Index: 397938 (C4 (Web Text))

Max Activating Token Index: 128

Text #16

Max Range: 0.0692. Min Range: -0.0692

Max Act: 0.0418. Min Act: -0.0002

Data Index: 1173193 (C4 (Web Text))

Max Activating Token Index: 920

Text #17

Max Range: 0.0692. Min Range: -0.0692

Max Act: 0.0446. Min Act: -0.0002

Data Index: 623526 (C4 (Web Text))

Max Activating Token Index: 610

Text #18

Max Range: 0.0692. Min Range: -0.0692

Max Act: 0.0435. Min Act: -0.0002

Data Index: 1237759 (C4 (Web Text))

Max Activating Token Index: 129

Text #19

Max Range: 0.0692. Min Range: -0.0692

Max Act: 0.0440. Min Act: -0.0002

Data Index: 1152385 (C4 (Web Text))

Max Activating Token Index: 382