Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2135 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.7489. Min Range: -0.7489

Max Act: 0.7489. Min Act: -0.0000

Data Index: 771922 (C4 (Web Text))

Max Activating Token Index: 880

Text #1

Max Range: 0.7489. Min Range: -0.7489

Max Act: 0.7425. Min Act: -0.0000

Data Index: 1349906 (C4 (Web Text))

Max Activating Token Index: 79

Text #2

Max Range: 0.7489. Min Range: -0.7489

Max Act: 0.6923. Min Act: -0.0000

Data Index: 361179 (C4 (Web Text))

Max Activating Token Index: 46

Text #3

Max Range: 0.7489. Min Range: -0.7489

Max Act: 0.6414. Min Act: -0.0000

Data Index: 1180357 (C4 (Web Text))

Max Activating Token Index: 850

Text #4

Max Range: 0.7489. Min Range: -0.7489

Max Act: 0.6154. Min Act: -0.0000

Data Index: 353765 (C4 (Web Text))

Max Activating Token Index: 339

Text #5

Max Range: 0.7489. Min Range: -0.7489

Max Act: 0.5863. Min Act: -0.0000

Data Index: 1175501 (C4 (Web Text))

Max Activating Token Index: 67

Text #6

Max Range: 0.7489. Min Range: -0.7489

Max Act: 0.5713. Min Act: -0.0000

Data Index: 445275 (C4 (Web Text))

Max Activating Token Index: 670

Text #7

Max Range: 0.7489. Min Range: -0.7489

Max Act: 0.5381. Min Act: -0.0000

Data Index: 1165611 (C4 (Web Text))

Max Activating Token Index: 567

Text #8

Max Range: 0.7489. Min Range: -0.7489

Max Act: 0.5660. Min Act: -0.0000

Data Index: 1097661 (C4 (Web Text))

Max Activating Token Index: 26

Text #9

Max Range: 0.7489. Min Range: -0.7489

Max Act: 0.5412. Min Act: -0.0000

Data Index: 280133 (C4 (Web Text))

Max Activating Token Index: 541

Text #10

Max Range: 0.7489. Min Range: -0.7489

Max Act: 0.5553. Min Act: -0.0000

Data Index: 4008 (C4 (Web Text))

Max Activating Token Index: 757

Text #11

Max Range: 0.7489. Min Range: -0.7489

Max Act: 0.5327. Min Act: -0.0000

Data Index: 204925 (C4 (Web Text))

Max Activating Token Index: 1020

Text #12

Max Range: 0.7489. Min Range: -0.7489

Max Act: 0.5311. Min Act: -0.0000

Data Index: 1310088 (C4 (Web Text))

Max Activating Token Index: 78

Text #13

Max Range: 0.7489. Min Range: -0.7489

Max Act: 0.5466. Min Act: -0.0000

Data Index: 332285 (C4 (Web Text))

Max Activating Token Index: 628

Text #14

Max Range: 0.7489. Min Range: -0.7489

Max Act: 0.5461. Min Act: -0.0000

Data Index: 20254 (C4 (Web Text))

Max Activating Token Index: 464

Text #15

Max Range: 0.7489. Min Range: -0.7489

Max Act: 0.5430. Min Act: 0.0000

Data Index: 169931 (C4 (Web Text))

Max Activating Token Index: 134

Text #16

Max Range: 0.7489. Min Range: -0.7489

Max Act: 0.5096. Min Act: -0.0000

Data Index: 819637 (C4 (Web Text))

Max Activating Token Index: 1016

Text #17

Max Range: 0.7489. Min Range: -0.7489

Max Act: 0.5238. Min Act: -0.0000

Data Index: 275138 (C4 (Web Text))

Max Activating Token Index: 36

Text #18

Max Range: 0.7489. Min Range: -0.7489

Max Act: 0.5265. Min Act: -0.0000

Data Index: 184828 (C4 (Web Text))

Max Activating Token Index: 52

Text #19

Max Range: 0.7489. Min Range: -0.7489

Max Act: 0.5201. Min Act: -0.0000

Data Index: 473257 (C4 (Web Text))

Max Activating Token Index: 645