Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1177 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.9309. Min Range: -0.9309

Max Act: 0.9309. Min Act: -0.0000

Data Index: 703464 (C4 (Web Text))

Max Activating Token Index: 181

Text #1

Max Range: 0.9309. Min Range: -0.9309

Max Act: 0.9191. Min Act: -0.0000

Data Index: 1112624 (C4 (Web Text))

Max Activating Token Index: 530

Text #2

Max Range: 0.9309. Min Range: -0.9309

Max Act: 0.7907. Min Act: -0.0000

Data Index: 1320160 (C4 (Web Text))

Max Activating Token Index: 824

Text #3

Max Range: 0.9309. Min Range: -0.9309

Max Act: 0.8064. Min Act: -0.0000

Data Index: 974905 (C4 (Web Text))

Max Activating Token Index: 490

Text #4

Max Range: 0.9309. Min Range: -0.9309

Max Act: 0.7405. Min Act: -0.0000

Data Index: 385043 (C4 (Web Text))

Max Activating Token Index: 351

Text #5

Max Range: 0.9309. Min Range: -0.9309

Max Act: 0.7363. Min Act: -0.0000

Data Index: 1005809 (C4 (Web Text))

Max Activating Token Index: 103

Text #6

Max Range: 0.9309. Min Range: -0.9309

Max Act: 0.7254. Min Act: -0.0000

Data Index: 1006499 (C4 (Web Text))

Max Activating Token Index: 124

Text #7

Max Range: 0.9309. Min Range: -0.9309

Max Act: 0.7297. Min Act: -0.0000

Data Index: 101497 (C4 (Web Text))

Max Activating Token Index: 375

Text #8

Max Range: 0.9309. Min Range: -0.9309

Max Act: 0.6801. Min Act: -0.0000

Data Index: 1147884 (C4 (Web Text))

Max Activating Token Index: 479

Text #9

Max Range: 0.9309. Min Range: -0.9309

Max Act: 0.7372. Min Act: -0.0000

Data Index: 155408 (C4 (Web Text))

Max Activating Token Index: 677

Text #10

Max Range: 0.9309. Min Range: -0.9309

Max Act: 0.7107. Min Act: -0.0000

Data Index: 497114 (C4 (Web Text))

Max Activating Token Index: 358

Text #11

Max Range: 0.9309. Min Range: -0.9309

Max Act: 0.6714. Min Act: -0.0000

Data Index: 296002 (C4 (Web Text))

Max Activating Token Index: 640

Text #12

Max Range: 0.9309. Min Range: -0.9309

Max Act: 0.6393. Min Act: -0.0000

Data Index: 1111946 (C4 (Web Text))

Max Activating Token Index: 64

Text #13

Max Range: 0.9309. Min Range: -0.9309

Max Act: 0.6419. Min Act: -0.0000

Data Index: 1219518 (C4 (Web Text))

Max Activating Token Index: 95

Text #14

Max Range: 0.9309. Min Range: -0.9309

Max Act: 0.6047. Min Act: -0.0000

Data Index: 1287962 (C4 (Web Text))

Max Activating Token Index: 771

Text #15

Max Range: 0.9309. Min Range: -0.9309

Max Act: 0.5882. Min Act: -0.0000

Data Index: 43093 (C4 (Web Text))

Max Activating Token Index: 922

Text #16

Max Range: 0.9309. Min Range: -0.9309

Max Act: 0.5815. Min Act: -0.0000

Data Index: 914540 (C4 (Web Text))

Max Activating Token Index: 441

Text #17

Max Range: 0.9309. Min Range: -0.9309

Max Act: 0.5870. Min Act: -0.0000

Data Index: 735513 (C4 (Web Text))

Max Activating Token Index: 291

Text #18

Max Range: 0.9309. Min Range: -0.9309

Max Act: 0.6190. Min Act: -0.0000

Data Index: 1269453 (C4 (Web Text))

Max Activating Token Index: 408

Text #19

Max Range: 0.9309. Min Range: -0.9309

Max Act: 0.5575. Min Act: -0.0000

Data Index: 1010512 (C4 (Web Text))

Max Activating Token Index: 689