Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3000 in Layer 1

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.9157. Min Range: -0.9157

Max Act: 0.8902. Min Act: -0.0003

Data Index: 355854 (C4 (Web Text))

Max Activating Token Index: 773

Text #1

Max Range: 0.9157. Min Range: -0.9157

Max Act: 0.9140. Min Act: -0.0003

Data Index: 1113144 (C4 (Web Text))

Max Activating Token Index: 796

Text #2

Max Range: 0.9157. Min Range: -0.9157

Max Act: 0.9157. Min Act: -0.0003

Data Index: 904241 (C4 (Web Text))

Max Activating Token Index: 696

Text #3

Max Range: 0.9157. Min Range: -0.9157

Max Act: 0.8086. Min Act: -0.0003

Data Index: 150596 (C4 (Web Text))

Max Activating Token Index: 818

Text #4

Max Range: 0.9157. Min Range: -0.9157

Max Act: 0.8263. Min Act: -0.0003

Data Index: 82036 (C4 (Web Text))

Max Activating Token Index: 861

Text #5

Max Range: 0.9157. Min Range: -0.9157

Max Act: 0.8233. Min Act: -0.0003

Data Index: 1100937 (C4 (Web Text))

Max Activating Token Index: 512

Text #6

Max Range: 0.9157. Min Range: -0.9157

Max Act: 0.8157. Min Act: -0.0002

Data Index: 1289516 (C4 (Web Text))

Max Activating Token Index: 560

Text #7

Max Range: 0.9157. Min Range: -0.9157

Max Act: 0.7794. Min Act: -0.0003

Data Index: 516252 (C4 (Web Text))

Max Activating Token Index: 26

Text #8

Max Range: 0.9157. Min Range: -0.9157

Max Act: 0.8035. Min Act: -0.0003

Data Index: 644591 (C4 (Web Text))

Max Activating Token Index: 488

Text #9

Max Range: 0.9157. Min Range: -0.9157

Max Act: 0.7987. Min Act: -0.0002

Data Index: 367271 (C4 (Web Text))

Max Activating Token Index: 262

Text #10

Max Range: 0.9157. Min Range: -0.9157

Max Act: 0.7770. Min Act: -0.0003

Data Index: 1193638 (C4 (Web Text))

Max Activating Token Index: 116

Text #11

Max Range: 0.9157. Min Range: -0.9157

Max Act: 0.7749. Min Act: -0.0003

Data Index: 510669 (C4 (Web Text))

Max Activating Token Index: 610

Text #12

Max Range: 0.9157. Min Range: -0.9157

Max Act: 0.7966. Min Act: -0.0003

Data Index: 290127 (C4 (Web Text))

Max Activating Token Index: 118

Text #13

Max Range: 0.9157. Min Range: -0.9157

Max Act: 0.7694. Min Act: -0.0002

Data Index: 1043381 (C4 (Web Text))

Max Activating Token Index: 129

Text #14

Max Range: 0.9157. Min Range: -0.9157

Max Act: 0.8364. Min Act: -0.0002

Data Index: 480902 (C4 (Web Text))

Max Activating Token Index: 949

Text #15

Max Range: 0.9157. Min Range: -0.9157

Max Act: 0.7632. Min Act: -0.0003

Data Index: 1197691 (C4 (Web Text))

Max Activating Token Index: 173

Text #16

Max Range: 0.9157. Min Range: -0.9157

Max Act: 0.7622. Min Act: -0.0003

Data Index: 475846 (C4 (Web Text))

Max Activating Token Index: 256

Text #17

Max Range: 0.9157. Min Range: -0.9157

Max Act: 0.7366. Min Act: -0.0003

Data Index: 629361 (C4 (Web Text))

Max Activating Token Index: 579

Text #18

Max Range: 0.9157. Min Range: -0.9157

Max Act: 0.7574. Min Act: -0.0003

Data Index: 926557 (C4 (Web Text))

Max Activating Token Index: 237

Text #19

Max Range: 0.9157. Min Range: -0.9157

Max Act: 0.7140. Min Act: -0.0003

Data Index: 1212958 (C4 (Web Text))

Max Activating Token Index: 260