Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 336 in Layer 1

Load this data into an Interactive Neuroscope

See Documentation here

Hooked Transformer Loading: HookedTransformer.from_pretrained('solu-12l')

Click the toggle to see the full text!

Text #0

Max Range: 2.6353. Min Range: -2.6353

Max Act: 2.6353. Min Act: -0.0002

Data Index: 367081 (C4 (Web Text))

Max Activating Token Index: 359

Text #1

Max Range: 2.6353. Min Range: -2.6353

Max Act: 2.4423. Min Act: -0.0002

Data Index: 734059 (C4 (Web Text))

Max Activating Token Index: 579

Text #2

Max Range: 2.6353. Min Range: -2.6353

Max Act: 2.3610. Min Act: -0.0002

Data Index: 847259 (C4 (Web Text))

Max Activating Token Index: 27

Text #3

Max Range: 2.6353. Min Range: -2.6353

Max Act: 2.3476. Min Act: -0.0003

Data Index: 1221224 (C4 (Web Text))

Max Activating Token Index: 700

Text #4

Max Range: 2.6353. Min Range: -2.6353

Max Act: 2.2438. Min Act: -0.0002

Data Index: 156696 (C4 (Web Text))

Max Activating Token Index: 81

Text #5

Max Range: 2.6353. Min Range: -2.6353

Max Act: 2.2317. Min Act: -0.0003

Data Index: 1302188 (C4 (Web Text))

Max Activating Token Index: 83

Text #6

Max Range: 2.6353. Min Range: -2.6353

Max Act: 2.2325. Min Act: -0.0002

Data Index: 213233 (C4 (Web Text))

Max Activating Token Index: 501

Text #7

Max Range: 2.6353. Min Range: -2.6353

Max Act: 2.2419. Min Act: -0.0002

Data Index: 1005653 (C4 (Web Text))

Max Activating Token Index: 150

Text #8

Max Range: 2.6353. Min Range: -2.6353

Max Act: 2.2463. Min Act: -0.0002

Data Index: 50765 (C4 (Web Text))

Max Activating Token Index: 594

Text #9

Max Range: 2.6353. Min Range: -2.6353

Max Act: 2.1924. Min Act: -0.0002

Data Index: 1056557 (C4 (Web Text))

Max Activating Token Index: 657

Text #10

Max Range: 2.6353. Min Range: -2.6353

Max Act: 2.1941. Min Act: -0.0002

Data Index: 209954 (C4 (Web Text))

Max Activating Token Index: 73

Text #11

Max Range: 2.6353. Min Range: -2.6353

Max Act: 2.1636. Min Act: -0.0002

Data Index: 918430 (C4 (Web Text))

Max Activating Token Index: 922

Text #12

Max Range: 2.6353. Min Range: -2.6353

Max Act: 2.1587. Min Act: -0.0002

Data Index: 1066209 (C4 (Web Text))

Max Activating Token Index: 306

Text #13

Max Range: 2.6353. Min Range: -2.6353

Max Act: 2.1824. Min Act: -0.0002

Data Index: 1169831 (C4 (Web Text))

Max Activating Token Index: 427

Text #14

Max Range: 2.6353. Min Range: -2.6353

Max Act: 2.2065. Min Act: -0.0002

Data Index: 1097123 (C4 (Web Text))

Max Activating Token Index: 363

Text #15

Max Range: 2.6353. Min Range: -2.6353

Max Act: 2.2203. Min Act: -0.0002

Data Index: 1206080 (C4 (Web Text))

Max Activating Token Index: 406

Text #16

Max Range: 2.6353. Min Range: -2.6353

Max Act: 2.2216. Min Act: -0.0002

Data Index: 1086029 (C4 (Web Text))

Max Activating Token Index: 43

Text #17

Max Range: 2.6353. Min Range: -2.6353

Max Act: 2.1616. Min Act: -0.0002

Data Index: 285195 (C4 (Web Text))

Max Activating Token Index: 48

Text #18

Max Range: 2.6353. Min Range: -2.6353

Max Act: 2.1323. Min Act: -0.0002

Data Index: 687500 (C4 (Web Text))

Max Activating Token Index: 343

Text #19

Max Range: 2.6353. Min Range: -2.6353

Max Act: 2.1577. Min Act: -0.0002

Data Index: 441926 (C4 (Web Text))

Max Activating Token Index: 867