Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1509 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 3.0286. Min Range: -3.0286

Max Act: 3.0286. Min Act: -0.0000

Data Index: 391337 (C4 (Web Text))

Max Activating Token Index: 970

Text #1

Max Range: 3.0286. Min Range: -3.0286

Max Act: 2.0892. Min Act: -0.0000

Data Index: 1030923 (C4 (Web Text))

Max Activating Token Index: 991

Text #2

Max Range: 3.0286. Min Range: -3.0286

Max Act: 2.0473. Min Act: -0.0000

Data Index: 1282706 (C4 (Web Text))

Max Activating Token Index: 756

Text #3

Max Range: 3.0286. Min Range: -3.0286

Max Act: 2.0811. Min Act: -0.0000

Data Index: 1163721 (C4 (Web Text))

Max Activating Token Index: 123

Text #4

Max Range: 3.0286. Min Range: -3.0286

Max Act: 1.8411. Min Act: -0.0000

Data Index: 456424 (C4 (Web Text))

Max Activating Token Index: 504

Text #5

Max Range: 3.0286. Min Range: -3.0286

Max Act: 1.9437. Min Act: -0.0000

Data Index: 650516 (C4 (Web Text))

Max Activating Token Index: 95

Text #6

Max Range: 3.0286. Min Range: -3.0286

Max Act: 1.7120. Min Act: -0.0000

Data Index: 731244 (C4 (Web Text))

Max Activating Token Index: 771

Text #7

Max Range: 3.0286. Min Range: -3.0286

Max Act: 1.7367. Min Act: -0.0000

Data Index: 1165252 (C4 (Web Text))

Max Activating Token Index: 346

Text #8

Max Range: 3.0286. Min Range: -3.0286

Max Act: 1.7374. Min Act: -0.0000

Data Index: 1289124 (C4 (Web Text))

Max Activating Token Index: 882

Text #9

Max Range: 3.0286. Min Range: -3.0286

Max Act: 1.7349. Min Act: -0.0000

Data Index: 65807 (C4 (Web Text))

Max Activating Token Index: 974

Text #10

Max Range: 3.0286. Min Range: -3.0286

Max Act: 1.7099. Min Act: -0.0000

Data Index: 140137 (C4 (Web Text))

Max Activating Token Index: 512

Text #11

Max Range: 3.0286. Min Range: -3.0286

Max Act: 1.8054. Min Act: -0.0000

Data Index: 1187539 (C4 (Web Text))

Max Activating Token Index: 779

Text #12

Max Range: 3.0286. Min Range: -3.0286

Max Act: 1.6601. Min Act: -0.0000

Data Index: 818653 (C4 (Web Text))

Max Activating Token Index: 976

Text #13

Max Range: 3.0286. Min Range: -3.0286

Max Act: 1.5670. Min Act: -0.0000

Data Index: 334959 (C4 (Web Text))

Max Activating Token Index: 567

Text #14

Max Range: 3.0286. Min Range: -3.0286

Max Act: 1.6379. Min Act: -0.0000

Data Index: 1068373 (C4 (Web Text))

Max Activating Token Index: 740

Text #15

Max Range: 3.0286. Min Range: -3.0286

Max Act: 1.6291. Min Act: -0.0000

Data Index: 256192 (C4 (Web Text))

Max Activating Token Index: 233

Text #16

Max Range: 3.0286. Min Range: -3.0286

Max Act: 1.7132. Min Act: -0.0000

Data Index: 231350 (C4 (Web Text))

Max Activating Token Index: 194

Text #17

Max Range: 3.0286. Min Range: -3.0286

Max Act: 1.6088. Min Act: -0.0000

Data Index: 1187166 (C4 (Web Text))

Max Activating Token Index: 1003

Text #18

Max Range: 3.0286. Min Range: -3.0286

Max Act: 1.6098. Min Act: -0.0000

Data Index: 369572 (C4 (Web Text))

Max Activating Token Index: 97

Text #19

Max Range: 3.0286. Min Range: -3.0286

Max Act: 1.6073. Min Act: -0.0000

Data Index: 61860 (C4 (Web Text))

Max Activating Token Index: 25