Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2265 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 5.7142. Min Range: -5.7142

Max Act: 5.7142. Min Act: -0.0000

Data Index: 84965 (C4 (Web Text))

Max Activating Token Index: 76

Text #1

Max Range: 5.7142. Min Range: -5.7142

Max Act: 5.5496. Min Act: -0.0000

Data Index: 931843 (C4 (Web Text))

Max Activating Token Index: 497

Text #2

Max Range: 5.7142. Min Range: -5.7142

Max Act: 4.7850. Min Act: -0.0000

Data Index: 391628 (C4 (Web Text))

Max Activating Token Index: 995

Text #3

Max Range: 5.7142. Min Range: -5.7142

Max Act: 4.8735. Min Act: -0.0000

Data Index: 776066 (C4 (Web Text))

Max Activating Token Index: 956

Text #4

Max Range: 5.7142. Min Range: -5.7142

Max Act: 4.5297. Min Act: -0.0000

Data Index: 72697 (C4 (Web Text))

Max Activating Token Index: 943

Text #5

Max Range: 5.7142. Min Range: -5.7142

Max Act: 4.4810. Min Act: -0.0000

Data Index: 1264307 (C4 (Web Text))

Max Activating Token Index: 992

Text #6

Max Range: 5.7142. Min Range: -5.7142

Max Act: 4.6174. Min Act: -0.0000

Data Index: 978212 (C4 (Web Text))

Max Activating Token Index: 336

Text #7

Max Range: 5.7142. Min Range: -5.7142

Max Act: 4.6022. Min Act: -0.0000

Data Index: 1092746 (C4 (Web Text))

Max Activating Token Index: 1010

Text #8

Max Range: 5.7142. Min Range: -5.7142

Max Act: 4.1961. Min Act: -0.0000

Data Index: 148902 (C4 (Web Text))

Max Activating Token Index: 759

Text #9

Max Range: 5.7142. Min Range: -5.7142

Max Act: 4.3498. Min Act: -0.0000

Data Index: 867587 (C4 (Web Text))

Max Activating Token Index: 249

Text #10

Max Range: 5.7142. Min Range: -5.7142

Max Act: 4.3502. Min Act: -0.0000

Data Index: 994629 (C4 (Web Text))

Max Activating Token Index: 895

Text #11

Max Range: 5.7142. Min Range: -5.7142

Max Act: 4.3280. Min Act: -0.0000

Data Index: 93174 (C4 (Web Text))

Max Activating Token Index: 374

Text #12

Max Range: 5.7142. Min Range: -5.7142

Max Act: 4.0101. Min Act: -0.0000

Data Index: 913611 (C4 (Web Text))

Max Activating Token Index: 38

Text #13

Max Range: 5.7142. Min Range: -5.7142

Max Act: 4.1971. Min Act: -0.0000

Data Index: 296332 (C4 (Web Text))

Max Activating Token Index: 119

Text #14

Max Range: 5.7142. Min Range: -5.7142

Max Act: 4.1676. Min Act: -0.0000

Data Index: 1081313 (C4 (Web Text))

Max Activating Token Index: 146

Text #15

Max Range: 5.7142. Min Range: -5.7142

Max Act: 4.1735. Min Act: -0.0000

Data Index: 44604 (C4 (Web Text))

Max Activating Token Index: 520

Text #16

Max Range: 5.7142. Min Range: -5.7142

Max Act: 4.3461. Min Act: -0.0000

Data Index: 343110 (C4 (Web Text))

Max Activating Token Index: 332

Text #17

Max Range: 5.7142. Min Range: -5.7142

Max Act: 4.1553. Min Act: -0.0000

Data Index: 780561 (C4 (Web Text))

Max Activating Token Index: 140

Text #18

Max Range: 5.7142. Min Range: -5.7142

Max Act: 3.9282. Min Act: -0.0000

Data Index: 1084551 (C4 (Web Text))

Max Activating Token Index: 45

Text #19

Max Range: 5.7142. Min Range: -5.7142

Max Act: 4.0795. Min Act: -0.0000

Data Index: 233444 (C4 (Web Text))

Max Activating Token Index: 704