Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1575 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 2.1799. Min Range: -2.1799

Max Act: 2.0279. Min Act: -0.0000

Data Index: 1028442 (C4 (Web Text))

Max Activating Token Index: 208

Text #1

Max Range: 2.1799. Min Range: -2.1799

Max Act: 1.9877. Min Act: -0.0000

Data Index: 644514 (C4 (Web Text))

Max Activating Token Index: 183

Text #2

Max Range: 2.1799. Min Range: -2.1799

Max Act: 2.1799. Min Act: -0.0000

Data Index: 334683 (C4 (Web Text))

Max Activating Token Index: 228

Text #3

Max Range: 2.1799. Min Range: -2.1799

Max Act: 1.9170. Min Act: -0.0000

Data Index: 962331 (C4 (Web Text))

Max Activating Token Index: 88

Text #4

Max Range: 2.1799. Min Range: -2.1799

Max Act: 1.8150. Min Act: -0.0000

Data Index: 797927 (C4 (Web Text))

Max Activating Token Index: 119

Text #5

Max Range: 2.1799. Min Range: -2.1799

Max Act: 1.7339. Min Act: -0.0000

Data Index: 49901 (C4 (Web Text))

Max Activating Token Index: 313

Text #6

Max Range: 2.1799. Min Range: -2.1799

Max Act: 1.8225. Min Act: -0.0000

Data Index: 1071511 (C4 (Web Text))

Max Activating Token Index: 384

Text #7

Max Range: 2.1799. Min Range: -2.1799

Max Act: 1.8092. Min Act: -0.0000

Data Index: 218721 (C4 (Web Text))

Max Activating Token Index: 833

Text #8

Max Range: 2.1799. Min Range: -2.1799

Max Act: 1.8098. Min Act: -0.0000

Data Index: 322771 (C4 (Web Text))

Max Activating Token Index: 70

Text #9

Max Range: 2.1799. Min Range: -2.1799

Max Act: 1.7095. Min Act: -0.0000

Data Index: 934960 (C4 (Web Text))

Max Activating Token Index: 503

Text #10

Max Range: 2.1799. Min Range: -2.1799

Max Act: 1.7882. Min Act: -0.0000

Data Index: 895568 (C4 (Web Text))

Max Activating Token Index: 929

Text #11

Max Range: 2.1799. Min Range: -2.1799

Max Act: 1.7740. Min Act: -0.0000

Data Index: 61704 (C4 (Web Text))

Max Activating Token Index: 508

Text #12

Max Range: 2.1799. Min Range: -2.1799

Max Act: 1.6637. Min Act: -0.0000

Data Index: 1135923 (C4 (Web Text))

Max Activating Token Index: 865

Text #13

Max Range: 2.1799. Min Range: -2.1799

Max Act: 1.7301. Min Act: -0.0000

Data Index: 432456 (C4 (Web Text))

Max Activating Token Index: 475

Text #14

Max Range: 2.1799. Min Range: -2.1799

Max Act: 1.7236. Min Act: -0.0000

Data Index: 101049 (C4 (Web Text))

Max Activating Token Index: 624

Text #15

Max Range: 2.1799. Min Range: -2.1799

Max Act: 1.6299. Min Act: -0.0000

Data Index: 807500 (C4 (Web Text))

Max Activating Token Index: 516

Text #16

Max Range: 2.1799. Min Range: -2.1799

Max Act: 1.6325. Min Act: -0.0000

Data Index: 165578 (C4 (Web Text))

Max Activating Token Index: 97

Text #17

Max Range: 2.1799. Min Range: -2.1799

Max Act: 1.6121. Min Act: -0.0000

Data Index: 921599 (C4 (Web Text))

Max Activating Token Index: 258

Text #18

Max Range: 2.1799. Min Range: -2.1799

Max Act: 1.7068. Min Act: -0.0000

Data Index: 143884 (C4 (Web Text))

Max Activating Token Index: 76

Text #19

Max Range: 2.1799. Min Range: -2.1799

Max Act: 1.7561. Min Act: -0.0000

Data Index: 1153171 (C4 (Web Text))

Max Activating Token Index: 752