Model: SoLU Model: 8 Layers, 4096 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1958 in Layer 3

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-8l')

Text #0

Max Range: 0.6144. Min Range: -0.6144

Max Act: 0.5939. Min Act: -0.0001

Data Index: 605745 (C4 (Web Text))

Max Activating Token Index: 923

Text #1

Max Range: 0.6144. Min Range: -0.6144

Max Act: 0.5325. Min Act: -0.0001

Data Index: 425949 (C4 (Web Text))

Max Activating Token Index: 696

Text #2

Max Range: 0.6144. Min Range: -0.6144

Max Act: 0.5438. Min Act: -0.0001

Data Index: 734383 (C4 (Web Text))

Max Activating Token Index: 544

Text #3

Max Range: 0.6144. Min Range: -0.6144

Max Act: 0.5613. Min Act: -0.0001

Data Index: 1334489 (C4 (Web Text))

Max Activating Token Index: 403

Text #4

Max Range: 0.6144. Min Range: -0.6144

Max Act: 0.6144. Min Act: -0.0001

Data Index: 5493 (C4 (Web Text))

Max Activating Token Index: 581

Text #5

Max Range: 0.6144. Min Range: -0.6144

Max Act: 0.5593. Min Act: -0.0001

Data Index: 931691 (C4 (Web Text))

Max Activating Token Index: 978

Text #6

Max Range: 0.6144. Min Range: -0.6144

Max Act: 0.5213. Min Act: -0.0001

Data Index: 425163 (C4 (Web Text))

Max Activating Token Index: 859

Text #7

Max Range: 0.6144. Min Range: -0.6144

Max Act: 0.5445. Min Act: -0.0001

Data Index: 143567 (C4 (Web Text))

Max Activating Token Index: 682

Text #8

Max Range: 0.6144. Min Range: -0.6144

Max Act: 0.5256. Min Act: -0.0001

Data Index: 1022975 (C4 (Web Text))

Max Activating Token Index: 978

Text #9

Max Range: 0.6144. Min Range: -0.6144

Max Act: 0.5302. Min Act: -0.0001

Data Index: 369382 (C4 (Web Text))

Max Activating Token Index: 578

Text #10

Max Range: 0.6144. Min Range: -0.6144

Max Act: 0.4953. Min Act: -0.0001

Data Index: 193323 (C4 (Web Text))

Max Activating Token Index: 799

Text #11

Max Range: 0.6144. Min Range: -0.6144

Max Act: 0.4991. Min Act: -0.0001

Data Index: 981338 (C4 (Web Text))

Max Activating Token Index: 207

Text #12

Max Range: 0.6144. Min Range: -0.6144

Max Act: 0.5064. Min Act: -0.0001

Data Index: 1325774 (C4 (Web Text))

Max Activating Token Index: 730

Text #13

Max Range: 0.6144. Min Range: -0.6144

Max Act: 0.5233. Min Act: -0.0001

Data Index: 1267046 (C4 (Web Text))

Max Activating Token Index: 850

Text #14

Max Range: 0.6144. Min Range: -0.6144

Max Act: 0.5249. Min Act: -0.0001

Data Index: 1287533 (C4 (Web Text))

Max Activating Token Index: 7

Text #15

Max Range: 0.6144. Min Range: -0.6144

Max Act: 0.5398. Min Act: -0.0001

Data Index: 604500 (C4 (Web Text))

Max Activating Token Index: 1002

Text #16

Max Range: 0.6144. Min Range: -0.6144

Max Act: 0.5043. Min Act: -0.0001

Data Index: 1136046 (C4 (Web Text))

Max Activating Token Index: 853

Text #17

Max Range: 0.6144. Min Range: -0.6144

Max Act: 0.4670. Min Act: -0.0001

Data Index: 965982 (C4 (Web Text))

Max Activating Token Index: 770

Text #18

Max Range: 0.6144. Min Range: -0.6144

Max Act: 0.4956. Min Act: -0.0001

Data Index: 490844 (C4 (Web Text))

Max Activating Token Index: 268

Text #19

Max Range: 0.6144. Min Range: -0.6144

Max Act: 0.5272. Min Act: -0.0001

Data Index: 819763 (C4 (Web Text))

Max Activating Token Index: 296