Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3328 in Layer 3

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.7804. Min Range: -0.7804

Max Act: 0.7804. Min Act: -0.0001

Data Index: 1276900 (C4 (Web Text))

Max Activating Token Index: 995

Text #1

Max Range: 0.7804. Min Range: -0.7804

Max Act: 0.7036. Min Act: -0.0001

Data Index: 726755 (C4 (Web Text))

Max Activating Token Index: 639

Text #2

Max Range: 0.7804. Min Range: -0.7804

Max Act: 0.6100. Min Act: -0.0001

Data Index: 450212 (C4 (Web Text))

Max Activating Token Index: 1010

Text #3

Max Range: 0.7804. Min Range: -0.7804

Max Act: 0.6039. Min Act: -0.0001

Data Index: 53863 (C4 (Web Text))

Max Activating Token Index: 21

Text #4

Max Range: 0.7804. Min Range: -0.7804

Max Act: 0.5671. Min Act: -0.0001

Data Index: 28386 (C4 (Web Text))

Max Activating Token Index: 3

Text #5

Max Range: 0.7804. Min Range: -0.7804

Max Act: 0.5639. Min Act: -0.0001

Data Index: 1057350 (C4 (Web Text))

Max Activating Token Index: 199

Text #6

Max Range: 0.7804. Min Range: -0.7804

Max Act: 0.5484. Min Act: -0.0001

Data Index: 668995 (C4 (Web Text))

Max Activating Token Index: 656

Text #7

Max Range: 0.7804. Min Range: -0.7804

Max Act: 0.5548. Min Act: -0.0001

Data Index: 595389 (C4 (Web Text))

Max Activating Token Index: 762

Text #8

Max Range: 0.7804. Min Range: -0.7804

Max Act: 0.5277. Min Act: -0.0001

Data Index: 29820 (C4 (Web Text))

Max Activating Token Index: 414

Text #9

Max Range: 0.7804. Min Range: -0.7804

Max Act: 0.5675. Min Act: -0.0001

Data Index: 489173 (C4 (Web Text))

Max Activating Token Index: 755

Text #10

Max Range: 0.7804. Min Range: -0.7804

Max Act: 0.5997. Min Act: -0.0001

Data Index: 1092671 (C4 (Web Text))

Max Activating Token Index: 69

Text #11

Max Range: 0.7804. Min Range: -0.7804

Max Act: 0.5640. Min Act: -0.0001

Data Index: 168412 (C4 (Web Text))

Max Activating Token Index: 521

Text #12

Max Range: 0.7804. Min Range: -0.7804

Max Act: 0.5134. Min Act: -0.0001

Data Index: 797056 (C4 (Web Text))

Max Activating Token Index: 33

Text #13

Max Range: 0.7804. Min Range: -0.7804

Max Act: 0.5450. Min Act: -0.0001

Data Index: 1295665 (C4 (Web Text))

Max Activating Token Index: 216

Text #14

Max Range: 0.7804. Min Range: -0.7804

Max Act: 0.5602. Min Act: -0.0001

Data Index: 434306 (C4 (Web Text))

Max Activating Token Index: 11

Text #15

Max Range: 0.7804. Min Range: -0.7804

Max Act: 0.5942. Min Act: -0.0001

Data Index: 1036715 (C4 (Web Text))

Max Activating Token Index: 58

Text #16

Max Range: 0.7804. Min Range: -0.7804

Max Act: 0.5361. Min Act: -0.0001

Data Index: 571001 (C4 (Web Text))

Max Activating Token Index: 1021

Text #17

Max Range: 0.7804. Min Range: -0.7804

Max Act: 0.5229. Min Act: -0.0001

Data Index: 638334 (C4 (Web Text))

Max Activating Token Index: 118

Text #18

Max Range: 0.7804. Min Range: -0.7804

Max Act: 0.5830. Min Act: -0.0001

Data Index: 431277 (C4 (Web Text))

Max Activating Token Index: 327

Text #19

Max Range: 0.7804. Min Range: -0.7804

Max Act: 0.5511. Min Act: -0.0001

Data Index: 1194769 (C4 (Web Text))

Max Activating Token Index: 422