Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4837 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.1418. Min Range: -1.1418

Max Act: 1.1333. Min Act: -0.0000

Data Index: 1033366 (C4 (Web Text))

Max Activating Token Index: 260

Text #1

Max Range: 1.1418. Min Range: -1.1418

Max Act: 1.1027. Min Act: -0.0000

Data Index: 1152934 (C4 (Web Text))

Max Activating Token Index: 355

Text #2

Max Range: 1.1418. Min Range: -1.1418

Max Act: 1.1418. Min Act: -0.0000

Data Index: 1025044 (C4 (Web Text))

Max Activating Token Index: 495

Text #3

Max Range: 1.1418. Min Range: -1.1418

Max Act: 0.9919. Min Act: -0.0000

Data Index: 31824 (C4 (Web Text))

Max Activating Token Index: 315

Text #4

Max Range: 1.1418. Min Range: -1.1418

Max Act: 0.9368. Min Act: -0.0000

Data Index: 1294033 (C4 (Web Text))

Max Activating Token Index: 321

Text #5

Max Range: 1.1418. Min Range: -1.1418

Max Act: 0.9685. Min Act: -0.0000

Data Index: 662743 (C4 (Web Text))

Max Activating Token Index: 117

Text #6

Max Range: 1.1418. Min Range: -1.1418

Max Act: 0.9530. Min Act: -0.0000

Data Index: 1118839 (C4 (Web Text))

Max Activating Token Index: 290

Text #7

Max Range: 1.1418. Min Range: -1.1418

Max Act: 0.9384. Min Act: -0.0000

Data Index: 764029 (C4 (Web Text))

Max Activating Token Index: 658

Text #8

Max Range: 1.1418. Min Range: -1.1418

Max Act: 0.9003. Min Act: -0.0000

Data Index: 856166 (C4 (Web Text))

Max Activating Token Index: 89

Text #9

Max Range: 1.1418. Min Range: -1.1418

Max Act: 0.9219. Min Act: -0.0000

Data Index: 992430 (C4 (Web Text))

Max Activating Token Index: 242

Text #10

Max Range: 1.1418. Min Range: -1.1418

Max Act: 0.9188. Min Act: -0.0000

Data Index: 618956 (C4 (Web Text))

Max Activating Token Index: 175

Text #11

Max Range: 1.1418. Min Range: -1.1418

Max Act: 0.8851. Min Act: -0.0000

Data Index: 571037 (C4 (Web Text))

Max Activating Token Index: 352

Text #12

Max Range: 1.1418. Min Range: -1.1418

Max Act: 0.8627. Min Act: -0.0000

Data Index: 725574 (C4 (Web Text))

Max Activating Token Index: 459

Text #13

Max Range: 1.1418. Min Range: -1.1418

Max Act: 0.8938. Min Act: -0.0000

Data Index: 1207368 (C4 (Web Text))

Max Activating Token Index: 638

Text #14

Max Range: 1.1418. Min Range: -1.1418

Max Act: 0.8615. Min Act: -0.0000

Data Index: 1165111 (C4 (Web Text))

Max Activating Token Index: 424

Text #15

Max Range: 1.1418. Min Range: -1.1418

Max Act: 0.8994. Min Act: -0.0000

Data Index: 190556 (C4 (Web Text))

Max Activating Token Index: 17

Text #16

Max Range: 1.1418. Min Range: -1.1418

Max Act: 0.9000. Min Act: -0.0000

Data Index: 130006 (C4 (Web Text))

Max Activating Token Index: 362

Text #17

Max Range: 1.1418. Min Range: -1.1418

Max Act: 0.9244. Min Act: -0.0000

Data Index: 353004 (C4 (Web Text))

Max Activating Token Index: 421

Text #18

Max Range: 1.1418. Min Range: -1.1418

Max Act: 0.8412. Min Act: -0.0000

Data Index: 925519 (C4 (Web Text))

Max Activating Token Index: 702

Text #19

Max Range: 1.1418. Min Range: -1.1418

Max Act: 0.8344. Min Act: -0.0000

Data Index: 535620 (C4 (Web Text))

Max Activating Token Index: 221