Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3086 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 0.8513. Min Range: -0.8513

Max Act: 0.8487. Min Act: -0.0000

Data Index: 682824 (C4 (Web Text))

Max Activating Token Index: 278

Text #1

Max Range: 0.8513. Min Range: -0.8513

Max Act: 0.8513. Min Act: -0.0000

Data Index: 799543 (C4 (Web Text))

Max Activating Token Index: 617

Text #2

Max Range: 0.8513. Min Range: -0.8513

Max Act: 0.6969. Min Act: -0.0000

Data Index: 6508 (C4 (Web Text))

Max Activating Token Index: 839

Text #3

Max Range: 0.8513. Min Range: -0.8513

Max Act: 0.7339. Min Act: -0.0000

Data Index: 527259 (C4 (Web Text))

Max Activating Token Index: 745

Text #4

Max Range: 0.8513. Min Range: -0.8513

Max Act: 0.7240. Min Act: -0.0000

Data Index: 1151801 (C4 (Web Text))

Max Activating Token Index: 340

Text #5

Max Range: 0.8513. Min Range: -0.8513

Max Act: 0.5733. Min Act: -0.0000

Data Index: 690103 (C4 (Web Text))

Max Activating Token Index: 495

Text #6

Max Range: 0.8513. Min Range: -0.8513

Max Act: 0.6424. Min Act: -0.0000

Data Index: 907066 (C4 (Web Text))

Max Activating Token Index: 801

Text #7

Max Range: 0.8513. Min Range: -0.8513

Max Act: 0.6568. Min Act: -0.0000

Data Index: 1188317 (C4 (Web Text))

Max Activating Token Index: 1016

Text #8

Max Range: 0.8513. Min Range: -0.8513

Max Act: 0.6629. Min Act: -0.0000

Data Index: 35946 (C4 (Web Text))

Max Activating Token Index: 716

Text #9

Max Range: 0.8513. Min Range: -0.8513

Max Act: 0.6164. Min Act: -0.0000

Data Index: 619529 (C4 (Web Text))

Max Activating Token Index: 765

Text #10

Max Range: 0.8513. Min Range: -0.8513

Max Act: 0.6354. Min Act: -0.0000

Data Index: 1197403 (C4 (Web Text))

Max Activating Token Index: 979

Text #11

Max Range: 0.8513. Min Range: -0.8513

Max Act: 0.6332. Min Act: -0.0000

Data Index: 517297 (C4 (Web Text))

Max Activating Token Index: 966

Text #12

Max Range: 0.8513. Min Range: -0.8513

Max Act: 0.6156. Min Act: -0.0000

Data Index: 1555147 (Python Code)

Max Activating Token Index: 996

Text #13

Max Range: 0.8513. Min Range: -0.8513

Max Act: 0.5806. Min Act: -0.0000

Data Index: 700275 (C4 (Web Text))

Max Activating Token Index: 772

Text #14

Max Range: 0.8513. Min Range: -0.8513

Max Act: 0.5948. Min Act: -0.0000

Data Index: 1250646 (C4 (Web Text))

Max Activating Token Index: 331

Text #15

Max Range: 0.8513. Min Range: -0.8513

Max Act: 0.5843. Min Act: -0.0000

Data Index: 102707 (C4 (Web Text))

Max Activating Token Index: 975

Text #16

Max Range: 0.8513. Min Range: -0.8513

Max Act: 0.5563. Min Act: -0.0000

Data Index: 618269 (C4 (Web Text))

Max Activating Token Index: 267

Text #17

Max Range: 0.8513. Min Range: -0.8513

Max Act: 0.5933. Min Act: -0.0000

Data Index: 353216 (C4 (Web Text))

Max Activating Token Index: 832

Text #18

Max Range: 0.8513. Min Range: -0.8513

Max Act: 0.5540. Min Act: -0.0000

Data Index: 132455 (C4 (Web Text))

Max Activating Token Index: 998

Text #19

Max Range: 0.8513. Min Range: -0.8513

Max Act: 0.6075. Min Act: -0.0000

Data Index: 53967 (C4 (Web Text))

Max Activating Token Index: 767