Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 897 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 3.0648. Min Range: -3.0648

Max Act: 3.0648. Min Act: -0.0002

Data Index: 164656 (C4 (Web Text))

Max Activating Token Index: 203

Text #1

Max Range: 3.0648. Min Range: -3.0648

Max Act: 2.3302. Min Act: -0.0002

Data Index: 87545 (C4 (Web Text))

Max Activating Token Index: 1007

Text #2

Max Range: 3.0648. Min Range: -3.0648

Max Act: 2.3106. Min Act: -0.0002

Data Index: 1267974 (C4 (Web Text))

Max Activating Token Index: 66

Text #3

Max Range: 3.0648. Min Range: -3.0648

Max Act: 2.1008. Min Act: -0.0002

Data Index: 513142 (C4 (Web Text))

Max Activating Token Index: 119

Text #4

Max Range: 3.0648. Min Range: -3.0648

Max Act: 2.2595. Min Act: -0.0002

Data Index: 824010 (C4 (Web Text))

Max Activating Token Index: 15

Text #5

Max Range: 3.0648. Min Range: -3.0648

Max Act: 2.3309. Min Act: -0.0002

Data Index: 594603 (C4 (Web Text))

Max Activating Token Index: 200

Text #6

Max Range: 3.0648. Min Range: -3.0648

Max Act: 2.2244. Min Act: -0.0002

Data Index: 477403 (C4 (Web Text))

Max Activating Token Index: 224

Text #7

Max Range: 3.0648. Min Range: -3.0648

Max Act: 2.0847. Min Act: -0.0002

Data Index: 118808 (C4 (Web Text))

Max Activating Token Index: 990

Text #8

Max Range: 3.0648. Min Range: -3.0648

Max Act: 1.9674. Min Act: -0.0002

Data Index: 1047133 (C4 (Web Text))

Max Activating Token Index: 152

Text #9

Max Range: 3.0648. Min Range: -3.0648

Max Act: 2.2335. Min Act: -0.0002

Data Index: 429429 (C4 (Web Text))

Max Activating Token Index: 615

Text #10

Max Range: 3.0648. Min Range: -3.0648

Max Act: 2.0131. Min Act: -0.0002

Data Index: 681487 (C4 (Web Text))

Max Activating Token Index: 620

Text #11

Max Range: 3.0648. Min Range: -3.0648

Max Act: 2.0716. Min Act: -0.0002

Data Index: 824206 (C4 (Web Text))

Max Activating Token Index: 245

Text #12

Max Range: 3.0648. Min Range: -3.0648

Max Act: 2.1022. Min Act: -0.0002

Data Index: 121317 (C4 (Web Text))

Max Activating Token Index: 183

Text #13

Max Range: 3.0648. Min Range: -3.0648

Max Act: 2.2111. Min Act: -0.0002

Data Index: 1085122 (C4 (Web Text))

Max Activating Token Index: 141

Text #14

Max Range: 3.0648. Min Range: -3.0648

Max Act: 2.0874. Min Act: -0.0002

Data Index: 971897 (C4 (Web Text))

Max Activating Token Index: 462

Text #15

Max Range: 3.0648. Min Range: -3.0648

Max Act: 2.0067. Min Act: -0.0002

Data Index: 120549 (C4 (Web Text))

Max Activating Token Index: 25

Text #16

Max Range: 3.0648. Min Range: -3.0648

Max Act: 2.1101. Min Act: -0.0002

Data Index: 1058701 (C4 (Web Text))

Max Activating Token Index: 102

Text #17

Max Range: 3.0648. Min Range: -3.0648

Max Act: 2.0967. Min Act: -0.0002

Data Index: 150301 (C4 (Web Text))

Max Activating Token Index: 19

Text #18

Max Range: 3.0648. Min Range: -3.0648

Max Act: 2.0766. Min Act: -0.0002

Data Index: 703131 (C4 (Web Text))

Max Activating Token Index: 100

Text #19

Max Range: 3.0648. Min Range: -3.0648

Max Act: 2.2210. Min Act: -0.0002

Data Index: 1008688 (C4 (Web Text))

Max Activating Token Index: 52