Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 5036 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 3.7975. Min Range: -3.7975

Max Act: 3.6474. Min Act: -0.0000

Data Index: 419993 (C4 (Web Text))

Max Activating Token Index: 612

Text #1

Max Range: 3.7975. Min Range: -3.7975

Max Act: 3.3666. Min Act: -0.0000

Data Index: 111657 (C4 (Web Text))

Max Activating Token Index: 34

Text #2

Max Range: 3.7975. Min Range: -3.7975

Max Act: 3.4984. Min Act: -0.0000

Data Index: 624596 (C4 (Web Text))

Max Activating Token Index: 967

Text #3

Max Range: 3.7975. Min Range: -3.7975

Max Act: 3.5089. Min Act: -0.0000

Data Index: 1292186 (C4 (Web Text))

Max Activating Token Index: 982

Text #4

Max Range: 3.7975. Min Range: -3.7975

Max Act: 3.7975. Min Act: -0.0000

Data Index: 856179 (C4 (Web Text))

Max Activating Token Index: 267

Text #5

Max Range: 3.7975. Min Range: -3.7975

Max Act: 3.1955. Min Act: -0.0000

Data Index: 756344 (C4 (Web Text))

Max Activating Token Index: 35

Text #6

Max Range: 3.7975. Min Range: -3.7975

Max Act: 3.4908. Min Act: -0.0000

Data Index: 1129844 (C4 (Web Text))

Max Activating Token Index: 572

Text #7

Max Range: 3.7975. Min Range: -3.7975

Max Act: 3.4710. Min Act: -0.0000

Data Index: 1173175 (C4 (Web Text))

Max Activating Token Index: 106

Text #8

Max Range: 3.7975. Min Range: -3.7975

Max Act: 3.2751. Min Act: -0.0000

Data Index: 654985 (C4 (Web Text))

Max Activating Token Index: 602

Text #9

Max Range: 3.7975. Min Range: -3.7975

Max Act: 3.3366. Min Act: -0.0000

Data Index: 67203 (C4 (Web Text))

Max Activating Token Index: 603

Text #10

Max Range: 3.7975. Min Range: -3.7975

Max Act: 3.0160. Min Act: -0.0000

Data Index: 756192 (C4 (Web Text))

Max Activating Token Index: 428

Text #11

Max Range: 3.7975. Min Range: -3.7975

Max Act: 3.1722. Min Act: -0.0000

Data Index: 199281 (C4 (Web Text))

Max Activating Token Index: 492

Text #12

Max Range: 3.7975. Min Range: -3.7975

Max Act: 3.1416. Min Act: -0.0000

Data Index: 888554 (C4 (Web Text))

Max Activating Token Index: 167

Text #13

Max Range: 3.7975. Min Range: -3.7975

Max Act: 2.9725. Min Act: -0.0000

Data Index: 1033285 (C4 (Web Text))

Max Activating Token Index: 291

Text #14

Max Range: 3.7975. Min Range: -3.7975

Max Act: 3.1133. Min Act: -0.0000

Data Index: 730689 (C4 (Web Text))

Max Activating Token Index: 267

Text #15

Max Range: 3.7975. Min Range: -3.7975

Max Act: 3.0571. Min Act: -0.0000

Data Index: 357205 (C4 (Web Text))

Max Activating Token Index: 989

Text #16

Max Range: 3.7975. Min Range: -3.7975

Max Act: 3.0528. Min Act: -0.0000

Data Index: 47253 (C4 (Web Text))

Max Activating Token Index: 767

Text #17

Max Range: 3.7975. Min Range: -3.7975

Max Act: 3.0300. Min Act: -0.0000

Data Index: 285524 (C4 (Web Text))

Max Activating Token Index: 334

Text #18

Max Range: 3.7975. Min Range: -3.7975

Max Act: 3.1581. Min Act: -0.0000

Data Index: 958889 (C4 (Web Text))

Max Activating Token Index: 981

Text #19

Max Range: 3.7975. Min Range: -3.7975

Max Act: 2.9812. Min Act: -0.0000

Data Index: 1067118 (C4 (Web Text))

Max Activating Token Index: 639