Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1611 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 3.2200. Min Range: -3.2200

Max Act: 3.2200. Min Act: -0.0000

Data Index: 806551 (C4 (Web Text))

Max Activating Token Index: 264

Text #1

Max Range: 3.2200. Min Range: -3.2200

Max Act: 2.8223. Min Act: -0.0000

Data Index: 283767 (C4 (Web Text))

Max Activating Token Index: 141

Text #2

Max Range: 3.2200. Min Range: -3.2200

Max Act: 2.7928. Min Act: -0.0000

Data Index: 962832 (C4 (Web Text))

Max Activating Token Index: 622

Text #3

Max Range: 3.2200. Min Range: -3.2200

Max Act: 2.8090. Min Act: -0.0000

Data Index: 404793 (C4 (Web Text))

Max Activating Token Index: 169

Text #4

Max Range: 3.2200. Min Range: -3.2200

Max Act: 2.4027. Min Act: -0.0000

Data Index: 196148 (C4 (Web Text))

Max Activating Token Index: 728

Text #5

Max Range: 3.2200. Min Range: -3.2200

Max Act: 2.4319. Min Act: -0.0000

Data Index: 1093858 (C4 (Web Text))

Max Activating Token Index: 506

Text #6

Max Range: 3.2200. Min Range: -3.2200

Max Act: 2.4275. Min Act: -0.0000

Data Index: 614590 (C4 (Web Text))

Max Activating Token Index: 978

Text #7

Max Range: 3.2200. Min Range: -3.2200

Max Act: 2.3051. Min Act: -0.0000

Data Index: 1193251 (C4 (Web Text))

Max Activating Token Index: 359

Text #8

Max Range: 3.2200. Min Range: -3.2200

Max Act: 2.1834. Min Act: -0.0000

Data Index: 524236 (C4 (Web Text))

Max Activating Token Index: 596

Text #9

Max Range: 3.2200. Min Range: -3.2200

Max Act: 2.1543. Min Act: -0.0000

Data Index: 557023 (C4 (Web Text))

Max Activating Token Index: 442

Text #10

Max Range: 3.2200. Min Range: -3.2200

Max Act: 2.1181. Min Act: -0.0000

Data Index: 707379 (C4 (Web Text))

Max Activating Token Index: 775

Text #11

Max Range: 3.2200. Min Range: -3.2200

Max Act: 2.2190. Min Act: -0.0000

Data Index: 605380 (C4 (Web Text))

Max Activating Token Index: 203

Text #12

Max Range: 3.2200. Min Range: -3.2200

Max Act: 2.0977. Min Act: -0.0000

Data Index: 58375 (C4 (Web Text))

Max Activating Token Index: 475

Text #13

Max Range: 3.2200. Min Range: -3.2200

Max Act: 2.3085. Min Act: -0.0000

Data Index: 945454 (C4 (Web Text))

Max Activating Token Index: 566

Text #14

Max Range: 3.2200. Min Range: -3.2200

Max Act: 2.2031. Min Act: -0.0000

Data Index: 1035650 (C4 (Web Text))

Max Activating Token Index: 563

Text #15

Max Range: 3.2200. Min Range: -3.2200

Max Act: 2.2708. Min Act: -0.0000

Data Index: 1043239 (C4 (Web Text))

Max Activating Token Index: 310

Text #16

Max Range: 3.2200. Min Range: -3.2200

Max Act: 2.1562. Min Act: -0.0000

Data Index: 185049 (C4 (Web Text))

Max Activating Token Index: 354

Text #17

Max Range: 3.2200. Min Range: -3.2200

Max Act: 2.1202. Min Act: -0.0000

Data Index: 155250 (C4 (Web Text))

Max Activating Token Index: 1009

Text #18

Max Range: 3.2200. Min Range: -3.2200

Max Act: 2.1865. Min Act: -0.0000

Data Index: 688707 (C4 (Web Text))

Max Activating Token Index: 937

Text #19

Max Range: 3.2200. Min Range: -3.2200

Max Act: 2.2071. Min Act: -0.0000

Data Index: 700778 (C4 (Web Text))

Max Activating Token Index: 86