Model: SoLU Model: 8 Layers, 4096 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 597 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-8l')

Text #0

Max Range: 1.6787. Min Range: -1.6787

Max Act: 1.6787. Min Act: -0.0001

Data Index: 479966 (C4 (Web Text))

Max Activating Token Index: 200

Text #1

Max Range: 1.6787. Min Range: -1.6787

Max Act: 1.5704. Min Act: -0.0001

Data Index: 1240419 (C4 (Web Text))

Max Activating Token Index: 180

Text #2

Max Range: 1.6787. Min Range: -1.6787

Max Act: 1.4194. Min Act: -0.0001

Data Index: 929002 (C4 (Web Text))

Max Activating Token Index: 694

Text #3

Max Range: 1.6787. Min Range: -1.6787

Max Act: 1.3818. Min Act: -0.0001

Data Index: 610580 (C4 (Web Text))

Max Activating Token Index: 946

Text #4

Max Range: 1.6787. Min Range: -1.6787

Max Act: 1.3636. Min Act: -0.0001

Data Index: 1133395 (C4 (Web Text))

Max Activating Token Index: 866

Text #5

Max Range: 1.6787. Min Range: -1.6787

Max Act: 1.3292. Min Act: -0.0001

Data Index: 13126 (C4 (Web Text))

Max Activating Token Index: 826

Text #6

Max Range: 1.6787. Min Range: -1.6787

Max Act: 1.3657. Min Act: -0.0001

Data Index: 62296 (C4 (Web Text))

Max Activating Token Index: 334

Text #7

Max Range: 1.6787. Min Range: -1.6787

Max Act: 1.4395. Min Act: -0.0001

Data Index: 343350 (C4 (Web Text))

Max Activating Token Index: 349

Text #8

Max Range: 1.6787. Min Range: -1.6787

Max Act: 1.3554. Min Act: -0.0001

Data Index: 1049079 (C4 (Web Text))

Max Activating Token Index: 248

Text #9

Max Range: 1.6787. Min Range: -1.6787

Max Act: 1.3479. Min Act: -0.0001

Data Index: 223864 (C4 (Web Text))

Max Activating Token Index: 259

Text #10

Max Range: 1.6787. Min Range: -1.6787

Max Act: 1.3850. Min Act: -0.0001

Data Index: 21702 (C4 (Web Text))

Max Activating Token Index: 84

Text #11

Max Range: 1.6787. Min Range: -1.6787

Max Act: 1.3814. Min Act: -0.0001

Data Index: 176718 (C4 (Web Text))

Max Activating Token Index: 860

Text #12

Max Range: 1.6787. Min Range: -1.6787

Max Act: 1.3382. Min Act: -0.0001

Data Index: 166713 (C4 (Web Text))

Max Activating Token Index: 949

Text #13

Max Range: 1.6787. Min Range: -1.6787

Max Act: 1.3113. Min Act: -0.0001

Data Index: 914167 (C4 (Web Text))

Max Activating Token Index: 648

Text #14

Max Range: 1.6787. Min Range: -1.6787

Max Act: 1.3510. Min Act: -0.0001

Data Index: 288580 (C4 (Web Text))

Max Activating Token Index: 785

Text #15

Max Range: 1.6787. Min Range: -1.6787

Max Act: 1.2404. Min Act: -0.0001

Data Index: 24222 (C4 (Web Text))

Max Activating Token Index: 998

Text #16

Max Range: 1.6787. Min Range: -1.6787

Max Act: 1.2689. Min Act: -0.0001

Data Index: 106657 (C4 (Web Text))

Max Activating Token Index: 399

Text #17

Max Range: 1.6787. Min Range: -1.6787

Max Act: 1.2881. Min Act: -0.0001

Data Index: 1358810 (C4 (Web Text))

Max Activating Token Index: 409

Text #18

Max Range: 1.6787. Min Range: -1.6787

Max Act: 1.3231. Min Act: -0.0001

Data Index: 1030336 (C4 (Web Text))

Max Activating Token Index: 102

Text #19

Max Range: 1.6787. Min Range: -1.6787

Max Act: 1.2831. Min Act: -0.0001

Data Index: 956263 (C4 (Web Text))

Max Activating Token Index: 187