Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1624 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 2.3907. Min Range: -2.3907

Max Act: 2.3907. Min Act: -0.0000

Data Index: 295881 (C4 (Web Text))

Max Activating Token Index: 715

Text #1

Max Range: 2.3907. Min Range: -2.3907

Max Act: 2.0072. Min Act: -0.0000

Data Index: 307562 (C4 (Web Text))

Max Activating Token Index: 554

Text #2

Max Range: 2.3907. Min Range: -2.3907

Max Act: 2.1102. Min Act: -0.0000

Data Index: 785032 (C4 (Web Text))

Max Activating Token Index: 941

Text #3

Max Range: 2.3907. Min Range: -2.3907

Max Act: 2.0620. Min Act: -0.0000

Data Index: 72341 (C4 (Web Text))

Max Activating Token Index: 554

Text #4

Max Range: 2.3907. Min Range: -2.3907

Max Act: 2.0540. Min Act: -0.0000

Data Index: 1010009 (C4 (Web Text))

Max Activating Token Index: 643

Text #5

Max Range: 2.3907. Min Range: -2.3907

Max Act: 1.9396. Min Act: -0.0000

Data Index: 683135 (C4 (Web Text))

Max Activating Token Index: 345

Text #6

Max Range: 2.3907. Min Range: -2.3907

Max Act: 2.0975. Min Act: -0.0000

Data Index: 1153575 (C4 (Web Text))

Max Activating Token Index: 412

Text #7

Max Range: 2.3907. Min Range: -2.3907

Max Act: 2.0883. Min Act: -0.0000

Data Index: 660330 (C4 (Web Text))

Max Activating Token Index: 435

Text #8

Max Range: 2.3907. Min Range: -2.3907

Max Act: 1.9778. Min Act: -0.0000

Data Index: 530543 (C4 (Web Text))

Max Activating Token Index: 869

Text #9

Max Range: 2.3907. Min Range: -2.3907

Max Act: 1.9326. Min Act: -0.0000

Data Index: 846513 (C4 (Web Text))

Max Activating Token Index: 311

Text #10

Max Range: 2.3907. Min Range: -2.3907

Max Act: 1.9271. Min Act: -0.0000

Data Index: 859398 (C4 (Web Text))

Max Activating Token Index: 391

Text #11

Max Range: 2.3907. Min Range: -2.3907

Max Act: 1.9237. Min Act: -0.0000

Data Index: 797071 (C4 (Web Text))

Max Activating Token Index: 564

Text #12

Max Range: 2.3907. Min Range: -2.3907

Max Act: 1.7999. Min Act: -0.0000

Data Index: 164800 (C4 (Web Text))

Max Activating Token Index: 588

Text #13

Max Range: 2.3907. Min Range: -2.3907

Max Act: 1.8664. Min Act: -0.0000

Data Index: 827957 (C4 (Web Text))

Max Activating Token Index: 334

Text #14

Max Range: 2.3907. Min Range: -2.3907

Max Act: 1.9592. Min Act: -0.0000

Data Index: 927648 (C4 (Web Text))

Max Activating Token Index: 857

Text #15

Max Range: 2.3907. Min Range: -2.3907

Max Act: 1.9432. Min Act: -0.0000

Data Index: 1319558 (C4 (Web Text))

Max Activating Token Index: 268

Text #16

Max Range: 2.3907. Min Range: -2.3907

Max Act: 1.8247. Min Act: -0.0000

Data Index: 136778 (C4 (Web Text))

Max Activating Token Index: 761

Text #17

Max Range: 2.3907. Min Range: -2.3907

Max Act: 1.8262. Min Act: -0.0000

Data Index: 521387 (C4 (Web Text))

Max Activating Token Index: 885

Text #18

Max Range: 2.3907. Min Range: -2.3907

Max Act: 1.8206. Min Act: -0.0000

Data Index: 152522 (C4 (Web Text))

Max Activating Token Index: 84

Text #19

Max Range: 2.3907. Min Range: -2.3907

Max Act: 1.8050. Min Act: -0.0000

Data Index: 627017 (C4 (Web Text))

Max Activating Token Index: 987