Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4016 in Layer 11

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 2.8031. Min Range: -2.8031

Max Act: 2.8031. Min Act: -0.0000

Data Index: 187470 (C4 (Web Text))

Max Activating Token Index: 973

Text #1

Max Range: 2.8031. Min Range: -2.8031

Max Act: 2.6784. Min Act: -0.0000

Data Index: 74890 (C4 (Web Text))

Max Activating Token Index: 977

Text #2

Max Range: 2.8031. Min Range: -2.8031

Max Act: 2.6101. Min Act: -0.0000

Data Index: 1135813 (C4 (Web Text))

Max Activating Token Index: 666

Text #3

Max Range: 2.8031. Min Range: -2.8031

Max Act: 2.5146. Min Act: -0.0000

Data Index: 488139 (C4 (Web Text))

Max Activating Token Index: 551

Text #4

Max Range: 2.8031. Min Range: -2.8031

Max Act: 2.7724. Min Act: -0.0000

Data Index: 393436 (C4 (Web Text))

Max Activating Token Index: 1009

Text #5

Max Range: 2.8031. Min Range: -2.8031

Max Act: 2.5767. Min Act: -0.0000

Data Index: 526304 (C4 (Web Text))

Max Activating Token Index: 594

Text #6

Max Range: 2.8031. Min Range: -2.8031

Max Act: 2.4172. Min Act: -0.0000

Data Index: 719805 (C4 (Web Text))

Max Activating Token Index: 437

Text #7

Max Range: 2.8031. Min Range: -2.8031

Max Act: 2.4014. Min Act: -0.0000

Data Index: 199137 (C4 (Web Text))

Max Activating Token Index: 423

Text #8

Max Range: 2.8031. Min Range: -2.8031

Max Act: 2.5133. Min Act: -0.0000

Data Index: 567066 (C4 (Web Text))

Max Activating Token Index: 489

Text #9

Max Range: 2.8031. Min Range: -2.8031

Max Act: 2.5031. Min Act: -0.0000

Data Index: 523495 (C4 (Web Text))

Max Activating Token Index: 200

Text #10

Max Range: 2.8031. Min Range: -2.8031

Max Act: 2.3820. Min Act: -0.0000

Data Index: 806887 (C4 (Web Text))

Max Activating Token Index: 179

Text #11

Max Range: 2.8031. Min Range: -2.8031

Max Act: 2.3885. Min Act: -0.0000

Data Index: 778038 (C4 (Web Text))

Max Activating Token Index: 432

Text #12

Max Range: 2.8031. Min Range: -2.8031

Max Act: 2.2592. Min Act: -0.0000

Data Index: 11683 (C4 (Web Text))

Max Activating Token Index: 172

Text #13

Max Range: 2.8031. Min Range: -2.8031

Max Act: 2.4728. Min Act: -0.0000

Data Index: 497073 (C4 (Web Text))

Max Activating Token Index: 141

Text #14

Max Range: 2.8031. Min Range: -2.8031

Max Act: 2.0903. Min Act: -0.0000

Data Index: 789875 (C4 (Web Text))

Max Activating Token Index: 909

Text #15

Max Range: 2.8031. Min Range: -2.8031

Max Act: 2.2145. Min Act: -0.0000

Data Index: 1255094 (C4 (Web Text))

Max Activating Token Index: 772

Text #16

Max Range: 2.8031. Min Range: -2.8031

Max Act: 2.4380. Min Act: -0.0000

Data Index: 1357685 (C4 (Web Text))

Max Activating Token Index: 542

Text #17

Max Range: 2.8031. Min Range: -2.8031

Max Act: 2.2883. Min Act: -0.0000

Data Index: 304402 (C4 (Web Text))

Max Activating Token Index: 920

Text #18

Max Range: 2.8031. Min Range: -2.8031

Max Act: 2.4010. Min Act: -0.0000

Data Index: 1070328 (C4 (Web Text))

Max Activating Token Index: 676

Text #19

Max Range: 2.8031. Min Range: -2.8031

Max Act: 2.2790. Min Act: -0.0000

Data Index: 773437 (C4 (Web Text))

Max Activating Token Index: 490