Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4187 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 0.0709. Min Range: -0.0709

Max Act: 0.0709. Min Act: -0.0002

Data Index: 1119844 (C4 (Web Text))

Max Activating Token Index: 222

Text #1

Max Range: 0.0709. Min Range: -0.0709

Max Act: 0.0490. Min Act: -0.0002

Data Index: 1122232 (C4 (Web Text))

Max Activating Token Index: 494

Text #2

Max Range: 0.0709. Min Range: -0.0709

Max Act: 0.0480. Min Act: -0.0002

Data Index: 119010 (C4 (Web Text))

Max Activating Token Index: 88

Text #3

Max Range: 0.0709. Min Range: -0.0709

Max Act: 0.0448. Min Act: -0.0002

Data Index: 617311 (C4 (Web Text))

Max Activating Token Index: 127

Text #4

Max Range: 0.0709. Min Range: -0.0709

Max Act: 0.0462. Min Act: -0.0002

Data Index: 10348 (C4 (Web Text))

Max Activating Token Index: 510

Text #5

Max Range: 0.0709. Min Range: -0.0709

Max Act: 0.0456. Min Act: -0.0002

Data Index: 224377 (C4 (Web Text))

Max Activating Token Index: 136

Text #6

Max Range: 0.0709. Min Range: -0.0709

Max Act: 0.0447. Min Act: -0.0002

Data Index: 876278 (C4 (Web Text))

Max Activating Token Index: 45

Text #7

Max Range: 0.0709. Min Range: -0.0709

Max Act: 0.0415. Min Act: -0.0002

Data Index: 6104 (C4 (Web Text))

Max Activating Token Index: 682

Text #8

Max Range: 0.0709. Min Range: -0.0709

Max Act: 0.0414. Min Act: -0.0002

Data Index: 689309 (C4 (Web Text))

Max Activating Token Index: 238

Text #9

Max Range: 0.0709. Min Range: -0.0709

Max Act: 0.0415. Min Act: -0.0002

Data Index: 207971 (C4 (Web Text))

Max Activating Token Index: 195

Text #10

Max Range: 0.0709. Min Range: -0.0709

Max Act: 0.0396. Min Act: -0.0002

Data Index: 532160 (C4 (Web Text))

Max Activating Token Index: 383

Text #11

Max Range: 0.0709. Min Range: -0.0709

Max Act: 0.0367. Min Act: -0.0002

Data Index: 833461 (C4 (Web Text))

Max Activating Token Index: 400

Text #12

Max Range: 0.0709. Min Range: -0.0709

Max Act: 0.0377. Min Act: -0.0002

Data Index: 943292 (C4 (Web Text))

Max Activating Token Index: 742

Text #13

Max Range: 0.0709. Min Range: -0.0709

Max Act: 0.0375. Min Act: -0.0002

Data Index: 809405 (C4 (Web Text))

Max Activating Token Index: 239

Text #14

Max Range: 0.0709. Min Range: -0.0709

Max Act: 0.0368. Min Act: -0.0002

Data Index: 582715 (C4 (Web Text))

Max Activating Token Index: 609

Text #15

Max Range: 0.0709. Min Range: -0.0709

Max Act: 0.0365. Min Act: -0.0002

Data Index: 1138973 (C4 (Web Text))

Max Activating Token Index: 429

Text #16

Max Range: 0.0709. Min Range: -0.0709

Max Act: 0.0367. Min Act: -0.0002

Data Index: 977776 (C4 (Web Text))

Max Activating Token Index: 460

Text #17

Max Range: 0.0709. Min Range: -0.0709

Max Act: 0.0371. Min Act: -0.0002

Data Index: 1121475 (C4 (Web Text))

Max Activating Token Index: 195

Text #18

Max Range: 0.0709. Min Range: -0.0709

Max Act: 0.0364. Min Act: -0.0002

Data Index: 1128939 (C4 (Web Text))

Max Activating Token Index: 60

Text #19

Max Range: 0.0709. Min Range: -0.0709

Max Act: 0.0368. Min Act: -0.0002

Data Index: 849751 (C4 (Web Text))

Max Activating Token Index: 408