Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1971 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.9047. Min Range: -0.9047

Max Act: 0.9047. Min Act: -0.0000

Data Index: 1026115 (C4 (Web Text))

Max Activating Token Index: 761

Text #1

Max Range: 0.9047. Min Range: -0.9047

Max Act: 0.8494. Min Act: -0.0000

Data Index: 1105663 (C4 (Web Text))

Max Activating Token Index: 247

Text #2

Max Range: 0.9047. Min Range: -0.9047

Max Act: 0.8252. Min Act: -0.0000

Data Index: 109130 (C4 (Web Text))

Max Activating Token Index: 284

Text #3

Max Range: 0.9047. Min Range: -0.9047

Max Act: 0.8309. Min Act: -0.0000

Data Index: 1415973 (Python Code)

Max Activating Token Index: 812

Text #4

Max Range: 0.9047. Min Range: -0.9047

Max Act: 0.7578. Min Act: -0.0000

Data Index: 1060393 (C4 (Web Text))

Max Activating Token Index: 133

Text #5

Max Range: 0.9047. Min Range: -0.9047

Max Act: 0.7319. Min Act: -0.0000

Data Index: 611563 (C4 (Web Text))

Max Activating Token Index: 204

Text #6

Max Range: 0.9047. Min Range: -0.9047

Max Act: 0.7725. Min Act: -0.0000

Data Index: 492636 (C4 (Web Text))

Max Activating Token Index: 851

Text #7

Max Range: 0.9047. Min Range: -0.9047

Max Act: 0.7727. Min Act: -0.0000

Data Index: 855131 (C4 (Web Text))

Max Activating Token Index: 78

Text #8

Max Range: 0.9047. Min Range: -0.9047

Max Act: 0.6966. Min Act: -0.0000

Data Index: 637840 (C4 (Web Text))

Max Activating Token Index: 349

Text #9

Max Range: 0.9047. Min Range: -0.9047

Max Act: 0.7077. Min Act: -0.0000

Data Index: 710765 (C4 (Web Text))

Max Activating Token Index: 888

Text #10

Max Range: 0.9047. Min Range: -0.9047

Max Act: 0.6992. Min Act: -0.0000

Data Index: 1432821 (Python Code)

Max Activating Token Index: 579

Text #11

Max Range: 0.9047. Min Range: -0.9047

Max Act: 0.6754. Min Act: -0.0000

Data Index: 33809 (C4 (Web Text))

Max Activating Token Index: 564

Text #12

Max Range: 0.9047. Min Range: -0.9047

Max Act: 0.6683. Min Act: -0.0000

Data Index: 543290 (C4 (Web Text))

Max Activating Token Index: 1021

Text #13

Max Range: 0.9047. Min Range: -0.9047

Max Act: 0.6486. Min Act: -0.0000

Data Index: 388422 (C4 (Web Text))

Max Activating Token Index: 82

Text #14

Max Range: 0.9047. Min Range: -0.9047

Max Act: 0.6212. Min Act: -0.0000

Data Index: 149999 (C4 (Web Text))

Max Activating Token Index: 855

Text #15

Max Range: 0.9047. Min Range: -0.9047

Max Act: 0.6715. Min Act: -0.0000

Data Index: 1286080 (C4 (Web Text))

Max Activating Token Index: 920

Text #16

Max Range: 0.9047. Min Range: -0.9047

Max Act: 0.6232. Min Act: -0.0000

Data Index: 980951 (C4 (Web Text))

Max Activating Token Index: 6

Text #17

Max Range: 0.9047. Min Range: -0.9047

Max Act: 0.6232. Min Act: -0.0000

Data Index: 251658 (C4 (Web Text))

Max Activating Token Index: 6

Text #18

Max Range: 0.9047. Min Range: -0.9047

Max Act: 0.6564. Min Act: -0.0000

Data Index: 49791 (C4 (Web Text))

Max Activating Token Index: 558

Text #19

Max Range: 0.9047. Min Range: -0.9047

Max Act: 0.6510. Min Act: -0.0000

Data Index: 75277 (C4 (Web Text))

Max Activating Token Index: 545