Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 522 in Layer 4

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 3.8493. Min Range: -3.8493

Max Act: 3.8493. Min Act: -0.0001

Data Index: 1193594 (C4 (Web Text))

Max Activating Token Index: 363

Text #1

Max Range: 3.8493. Min Range: -3.8493

Max Act: 3.7317. Min Act: -0.0001

Data Index: 526590 (C4 (Web Text))

Max Activating Token Index: 594

Text #2

Max Range: 3.8493. Min Range: -3.8493

Max Act: 3.4918. Min Act: -0.0001

Data Index: 1228325 (C4 (Web Text))

Max Activating Token Index: 607

Text #3

Max Range: 3.8493. Min Range: -3.8493

Max Act: 3.4417. Min Act: -0.0001

Data Index: 584841 (C4 (Web Text))

Max Activating Token Index: 853

Text #4

Max Range: 3.8493. Min Range: -3.8493

Max Act: 2.9951. Min Act: -0.0001

Data Index: 405314 (C4 (Web Text))

Max Activating Token Index: 902

Text #5

Max Range: 3.8493. Min Range: -3.8493

Max Act: 3.3741. Min Act: -0.0001

Data Index: 611834 (C4 (Web Text))

Max Activating Token Index: 227

Text #6

Max Range: 3.8493. Min Range: -3.8493

Max Act: 3.1393. Min Act: -0.0001

Data Index: 12982 (C4 (Web Text))

Max Activating Token Index: 718

Text #7

Max Range: 3.8493. Min Range: -3.8493

Max Act: 2.9875. Min Act: -0.0001

Data Index: 932697 (C4 (Web Text))

Max Activating Token Index: 97

Text #8

Max Range: 3.8493. Min Range: -3.8493

Max Act: 2.8178. Min Act: -0.0001

Data Index: 740611 (C4 (Web Text))

Max Activating Token Index: 61

Text #9

Max Range: 3.8493. Min Range: -3.8493

Max Act: 2.9226. Min Act: -0.0001

Data Index: 1110302 (C4 (Web Text))

Max Activating Token Index: 611

Text #10

Max Range: 3.8493. Min Range: -3.8493

Max Act: 2.8881. Min Act: -0.0001

Data Index: 1027793 (C4 (Web Text))

Max Activating Token Index: 625

Text #11

Max Range: 3.8493. Min Range: -3.8493

Max Act: 3.0043. Min Act: -0.0001

Data Index: 1157412 (C4 (Web Text))

Max Activating Token Index: 85

Text #12

Max Range: 3.8493. Min Range: -3.8493

Max Act: 3.0505. Min Act: -0.0001

Data Index: 1122656 (C4 (Web Text))

Max Activating Token Index: 917

Text #13

Max Range: 3.8493. Min Range: -3.8493

Max Act: 2.8755. Min Act: -0.0001

Data Index: 903593 (C4 (Web Text))

Max Activating Token Index: 310

Text #14

Max Range: 3.8493. Min Range: -3.8493

Max Act: 2.9761. Min Act: -0.0001

Data Index: 150368 (C4 (Web Text))

Max Activating Token Index: 756

Text #15

Max Range: 3.8493. Min Range: -3.8493

Max Act: 2.6629. Min Act: -0.0001

Data Index: 735024 (C4 (Web Text))

Max Activating Token Index: 919

Text #16

Max Range: 3.8493. Min Range: -3.8493

Max Act: 2.4313. Min Act: -0.0001

Data Index: 454570 (C4 (Web Text))

Max Activating Token Index: 221

Text #17

Max Range: 3.8493. Min Range: -3.8493

Max Act: 2.6423. Min Act: -0.0001

Data Index: 1238395 (C4 (Web Text))

Max Activating Token Index: 113

Text #18

Max Range: 3.8493. Min Range: -3.8493

Max Act: 2.4933. Min Act: -0.0001

Data Index: 788683 (C4 (Web Text))

Max Activating Token Index: 691

Text #19

Max Range: 3.8493. Min Range: -3.8493

Max Act: 2.4473. Min Act: -0.0001

Data Index: 113089 (C4 (Web Text))

Max Activating Token Index: 391