Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3451 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.4659. Min Range: -0.4659

Max Act: 0.4659. Min Act: -0.0000

Data Index: 247511 (C4 (Web Text))

Max Activating Token Index: 874

Text #1

Max Range: 0.4659. Min Range: -0.4659

Max Act: 0.4577. Min Act: -0.0000

Data Index: 503541 (C4 (Web Text))

Max Activating Token Index: 935

Text #2

Max Range: 0.4659. Min Range: -0.4659

Max Act: 0.4476. Min Act: -0.0000

Data Index: 550802 (C4 (Web Text))

Max Activating Token Index: 895

Text #3

Max Range: 0.4659. Min Range: -0.4659

Max Act: 0.3912. Min Act: -0.0000

Data Index: 476166 (C4 (Web Text))

Max Activating Token Index: 838

Text #4

Max Range: 0.4659. Min Range: -0.4659

Max Act: 0.3206. Min Act: -0.0000

Data Index: 848395 (C4 (Web Text))

Max Activating Token Index: 124

Text #5

Max Range: 0.4659. Min Range: -0.4659

Max Act: 0.3486. Min Act: -0.0000

Data Index: 73646 (C4 (Web Text))

Max Activating Token Index: 552

Text #6

Max Range: 0.4659. Min Range: -0.4659

Max Act: 0.3325. Min Act: -0.0000

Data Index: 523290 (C4 (Web Text))

Max Activating Token Index: 154

Text #7

Max Range: 0.4659. Min Range: -0.4659

Max Act: 0.3403. Min Act: -0.0000

Data Index: 931456 (C4 (Web Text))

Max Activating Token Index: 829

Text #8

Max Range: 0.4659. Min Range: -0.4659

Max Act: 0.3326. Min Act: -0.0000

Data Index: 757772 (C4 (Web Text))

Max Activating Token Index: 232

Text #9

Max Range: 0.4659. Min Range: -0.4659

Max Act: 0.3407. Min Act: -0.0000

Data Index: 935851 (C4 (Web Text))

Max Activating Token Index: 264

Text #10

Max Range: 0.4659. Min Range: -0.4659

Max Act: 0.3642. Min Act: -0.0000

Data Index: 504259 (C4 (Web Text))

Max Activating Token Index: 162

Text #11

Max Range: 0.4659. Min Range: -0.4659

Max Act: 0.3296. Min Act: -0.0000

Data Index: 836888 (C4 (Web Text))

Max Activating Token Index: 953

Text #12

Max Range: 0.4659. Min Range: -0.4659

Max Act: 0.3289. Min Act: -0.0000

Data Index: 328663 (C4 (Web Text))

Max Activating Token Index: 229

Text #13

Max Range: 0.4659. Min Range: -0.4659

Max Act: 0.2951. Min Act: -0.0000

Data Index: 474639 (C4 (Web Text))

Max Activating Token Index: 246

Text #14

Max Range: 0.4659. Min Range: -0.4659

Max Act: 0.3260. Min Act: -0.0000

Data Index: 1277121 (C4 (Web Text))

Max Activating Token Index: 354

Text #15

Max Range: 0.4659. Min Range: -0.4659

Max Act: 0.3005. Min Act: -0.0000

Data Index: 1316009 (C4 (Web Text))

Max Activating Token Index: 897

Text #16

Max Range: 0.4659. Min Range: -0.4659

Max Act: 0.3203. Min Act: -0.0000

Data Index: 723133 (C4 (Web Text))

Max Activating Token Index: 945

Text #17

Max Range: 0.4659. Min Range: -0.4659

Max Act: 0.3292. Min Act: -0.0000

Data Index: 29169 (C4 (Web Text))

Max Activating Token Index: 132

Text #18

Max Range: 0.4659. Min Range: -0.4659

Max Act: 0.3105. Min Act: -0.0000

Data Index: 993502 (C4 (Web Text))

Max Activating Token Index: 673

Text #19

Max Range: 0.4659. Min Range: -0.4659

Max Act: 0.3063. Min Act: -0.0000

Data Index: 820 (C4 (Web Text))

Max Activating Token Index: 300