Model: SoLU Model: 8 Layers, 4096 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3688 in Layer 4

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-8l')

Text #0

Max Range: 1.1460. Min Range: -1.1460

Max Act: 1.1460. Min Act: -0.0001

Data Index: 832624 (C4 (Web Text))

Max Activating Token Index: 445

Text #1

Max Range: 1.1460. Min Range: -1.1460

Max Act: 1.0754. Min Act: -0.0001

Data Index: 1137124 (C4 (Web Text))

Max Activating Token Index: 499

Text #2

Max Range: 1.1460. Min Range: -1.1460

Max Act: 0.9533. Min Act: -0.0001

Data Index: 698823 (C4 (Web Text))

Max Activating Token Index: 970

Text #3

Max Range: 1.1460. Min Range: -1.1460

Max Act: 0.8442. Min Act: -0.0001

Data Index: 498671 (C4 (Web Text))

Max Activating Token Index: 908

Text #4

Max Range: 1.1460. Min Range: -1.1460

Max Act: 0.8426. Min Act: -0.0001

Data Index: 525997 (C4 (Web Text))

Max Activating Token Index: 181

Text #5

Max Range: 1.1460. Min Range: -1.1460

Max Act: 0.8101. Min Act: -0.0001

Data Index: 95669 (C4 (Web Text))

Max Activating Token Index: 760

Text #6

Max Range: 1.1460. Min Range: -1.1460

Max Act: 0.7111. Min Act: -0.0001

Data Index: 1114828 (C4 (Web Text))

Max Activating Token Index: 469

Text #7

Max Range: 1.1460. Min Range: -1.1460

Max Act: 0.7375. Min Act: -0.0001

Data Index: 383586 (C4 (Web Text))

Max Activating Token Index: 439

Text #8

Max Range: 1.1460. Min Range: -1.1460

Max Act: 0.7226. Min Act: -0.0001

Data Index: 826890 (C4 (Web Text))

Max Activating Token Index: 780

Text #9

Max Range: 1.1460. Min Range: -1.1460

Max Act: 0.6781. Min Act: -0.0001

Data Index: 224582 (C4 (Web Text))

Max Activating Token Index: 915

Text #10

Max Range: 1.1460. Min Range: -1.1460

Max Act: 0.7166. Min Act: -0.0001

Data Index: 27707 (C4 (Web Text))

Max Activating Token Index: 42

Text #11

Max Range: 1.1460. Min Range: -1.1460

Max Act: 0.7238. Min Act: -0.0001

Data Index: 822540 (C4 (Web Text))

Max Activating Token Index: 1012

Text #12

Max Range: 1.1460. Min Range: -1.1460

Max Act: 0.6993. Min Act: -0.0001

Data Index: 1282284 (C4 (Web Text))

Max Activating Token Index: 330

Text #13

Max Range: 1.1460. Min Range: -1.1460

Max Act: 0.6948. Min Act: -0.0001

Data Index: 1325766 (C4 (Web Text))

Max Activating Token Index: 669

Text #14

Max Range: 1.1460. Min Range: -1.1460

Max Act: 0.6891. Min Act: -0.0001

Data Index: 296510 (C4 (Web Text))

Max Activating Token Index: 288

Text #15

Max Range: 1.1460. Min Range: -1.1460

Max Act: 0.6419. Min Act: -0.0001

Data Index: 120291 (C4 (Web Text))

Max Activating Token Index: 723

Text #16

Max Range: 1.1460. Min Range: -1.1460

Max Act: 0.6794. Min Act: -0.0001

Data Index: 1317534 (C4 (Web Text))

Max Activating Token Index: 1019

Text #17

Max Range: 1.1460. Min Range: -1.1460

Max Act: 0.6583. Min Act: -0.0001

Data Index: 790634 (C4 (Web Text))

Max Activating Token Index: 238

Text #18

Max Range: 1.1460. Min Range: -1.1460

Max Act: 0.6474. Min Act: -0.0001

Data Index: 12961 (C4 (Web Text))

Max Activating Token Index: 700

Text #19

Max Range: 1.1460. Min Range: -1.1460

Max Act: 0.6225. Min Act: -0.0001

Data Index: 565775 (C4 (Web Text))

Max Activating Token Index: 644