Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 5907 in Layer 11

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 0.8943. Min Range: -0.8943

Max Act: 0.8943. Min Act: -0.0000

Data Index: 656931 (C4 (Web Text))

Max Activating Token Index: 941

Text #1

Max Range: 0.8943. Min Range: -0.8943

Max Act: 0.7795. Min Act: -0.0000

Data Index: 1562827 (Python Code)

Max Activating Token Index: 365

Text #2

Max Range: 0.8943. Min Range: -0.8943

Max Act: 0.7347. Min Act: -0.0000

Data Index: 1383995 (Python Code)

Max Activating Token Index: 980

Text #3

Max Range: 0.8943. Min Range: -0.8943

Max Act: 0.6811. Min Act: -0.0000

Data Index: 1407785 (Python Code)

Max Activating Token Index: 281

Text #4

Max Range: 0.8943. Min Range: -0.8943

Max Act: 0.7037. Min Act: -0.0000

Data Index: 1144935 (C4 (Web Text))

Max Activating Token Index: 187

Text #5

Max Range: 0.8943. Min Range: -0.8943

Max Act: 0.6647. Min Act: -0.0000

Data Index: 232982 (C4 (Web Text))

Max Activating Token Index: 219

Text #6

Max Range: 0.8943. Min Range: -0.8943

Max Act: 0.5731. Min Act: -0.0000

Data Index: 1100793 (C4 (Web Text))

Max Activating Token Index: 927

Text #7

Max Range: 0.8943. Min Range: -0.8943

Max Act: 0.6032. Min Act: -0.0000

Data Index: 891859 (C4 (Web Text))

Max Activating Token Index: 935

Text #8

Max Range: 0.8943. Min Range: -0.8943

Max Act: 0.6576. Min Act: -0.0000

Data Index: 436160 (C4 (Web Text))

Max Activating Token Index: 826

Text #9

Max Range: 0.8943. Min Range: -0.8943

Max Act: 0.5884. Min Act: -0.0000

Data Index: 1447820 (Python Code)

Max Activating Token Index: 569

Text #10

Max Range: 0.8943. Min Range: -0.8943

Max Act: 0.5798. Min Act: -0.0000

Data Index: 255853 (C4 (Web Text))

Max Activating Token Index: 435

Text #11

Max Range: 0.8943. Min Range: -0.8943

Max Act: 0.6559. Min Act: -0.0000

Data Index: 32627 (C4 (Web Text))

Max Activating Token Index: 336

Text #12

Max Range: 0.8943. Min Range: -0.8943

Max Act: 0.5241. Min Act: -0.0000

Data Index: 1386559 (Python Code)

Max Activating Token Index: 366

Text #13

Max Range: 0.8943. Min Range: -0.8943

Max Act: 0.5584. Min Act: -0.0000

Data Index: 790180 (C4 (Web Text))

Max Activating Token Index: 294

Text #14

Max Range: 0.8943. Min Range: -0.8943

Max Act: 0.5679. Min Act: -0.0000

Data Index: 701236 (C4 (Web Text))

Max Activating Token Index: 581

Text #15

Max Range: 0.8943. Min Range: -0.8943

Max Act: 0.5492. Min Act: -0.0000

Data Index: 1269395 (C4 (Web Text))

Max Activating Token Index: 489

Text #16

Max Range: 0.8943. Min Range: -0.8943

Max Act: 0.5310. Min Act: -0.0000

Data Index: 875421 (C4 (Web Text))

Max Activating Token Index: 676

Text #17

Max Range: 0.8943. Min Range: -0.8943

Max Act: 0.5391. Min Act: -0.0000

Data Index: 337426 (C4 (Web Text))

Max Activating Token Index: 971

Text #18

Max Range: 0.8943. Min Range: -0.8943

Max Act: 0.5083. Min Act: -0.0000

Data Index: 1088377 (C4 (Web Text))

Max Activating Token Index: 966

Text #19

Max Range: 0.8943. Min Range: -0.8943

Max Act: 0.5230. Min Act: -0.0000

Data Index: 542187 (C4 (Web Text))

Max Activating Token Index: 221