Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 336 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.1748. Min Range: -0.1748

Max Act: 0.1748. Min Act: -0.0000

Data Index: 1290489 (C4 (Web Text))

Max Activating Token Index: 996

Text #1

Max Range: 0.1748. Min Range: -0.1748

Max Act: 0.1680. Min Act: -0.0000

Data Index: 218916 (C4 (Web Text))

Max Activating Token Index: 522

Text #2

Max Range: 0.1748. Min Range: -0.1748

Max Act: 0.1373. Min Act: -0.0000

Data Index: 1036826 (C4 (Web Text))

Max Activating Token Index: 745

Text #3

Max Range: 0.1748. Min Range: -0.1748

Max Act: 0.1317. Min Act: -0.0000

Data Index: 1029856 (C4 (Web Text))

Max Activating Token Index: 216

Text #4

Max Range: 0.1748. Min Range: -0.1748

Max Act: 0.1251. Min Act: -0.0000

Data Index: 310203 (C4 (Web Text))

Max Activating Token Index: 629

Text #5

Max Range: 0.1748. Min Range: -0.1748

Max Act: 0.1364. Min Act: -0.0000

Data Index: 711332 (C4 (Web Text))

Max Activating Token Index: 146

Text #6

Max Range: 0.1748. Min Range: -0.1748

Max Act: 0.1318. Min Act: -0.0000

Data Index: 995449 (C4 (Web Text))

Max Activating Token Index: 206

Text #7

Max Range: 0.1748. Min Range: -0.1748

Max Act: 0.1276. Min Act: -0.0000

Data Index: 891827 (C4 (Web Text))

Max Activating Token Index: 957

Text #8

Max Range: 0.1748. Min Range: -0.1748

Max Act: 0.1232. Min Act: -0.0000

Data Index: 458823 (C4 (Web Text))

Max Activating Token Index: 724

Text #9

Max Range: 0.1748. Min Range: -0.1748

Max Act: 0.1233. Min Act: -0.0000

Data Index: 788683 (C4 (Web Text))

Max Activating Token Index: 480

Text #10

Max Range: 0.1748. Min Range: -0.1748

Max Act: 0.1232. Min Act: -0.0000

Data Index: 358939 (C4 (Web Text))

Max Activating Token Index: 65

Text #11

Max Range: 0.1748. Min Range: -0.1748

Max Act: 0.1276. Min Act: -0.0000

Data Index: 321505 (C4 (Web Text))

Max Activating Token Index: 613

Text #12

Max Range: 0.1748. Min Range: -0.1748

Max Act: 0.1261. Min Act: -0.0000

Data Index: 808206 (C4 (Web Text))

Max Activating Token Index: 768

Text #13

Max Range: 0.1748. Min Range: -0.1748

Max Act: 0.1206. Min Act: -0.0000

Data Index: 592241 (C4 (Web Text))

Max Activating Token Index: 950

Text #14

Max Range: 0.1748. Min Range: -0.1748

Max Act: 0.1211. Min Act: -0.0000

Data Index: 1055524 (C4 (Web Text))

Max Activating Token Index: 991

Text #15

Max Range: 0.1748. Min Range: -0.1748

Max Act: 0.1243. Min Act: -0.0000

Data Index: 857378 (C4 (Web Text))

Max Activating Token Index: 49

Text #16

Max Range: 0.1748. Min Range: -0.1748

Max Act: 0.1202. Min Act: -0.0000

Data Index: 659468 (C4 (Web Text))

Max Activating Token Index: 236

Text #17

Max Range: 0.1748. Min Range: -0.1748

Max Act: 0.1244. Min Act: -0.0000

Data Index: 437685 (C4 (Web Text))

Max Activating Token Index: 502

Text #18

Max Range: 0.1748. Min Range: -0.1748

Max Act: 0.1136. Min Act: -0.0000

Data Index: 969315 (C4 (Web Text))

Max Activating Token Index: 400

Text #19

Max Range: 0.1748. Min Range: -0.1748

Max Act: 0.1122. Min Act: -0.0000

Data Index: 366843 (C4 (Web Text))

Max Activating Token Index: 812