Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 954 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 4.2941. Min Range: -4.2941

Max Act: 3.9770. Min Act: -0.0000

Data Index: 1224056 (C4 (Web Text))

Max Activating Token Index: 346

Text #1

Max Range: 4.2941. Min Range: -4.2941

Max Act: 4.2941. Min Act: -0.0000

Data Index: 311320 (C4 (Web Text))

Max Activating Token Index: 467

Text #2

Max Range: 4.2941. Min Range: -4.2941

Max Act: 3.8205. Min Act: -0.0000

Data Index: 625170 (C4 (Web Text))

Max Activating Token Index: 710

Text #3

Max Range: 4.2941. Min Range: -4.2941

Max Act: 3.6913. Min Act: -0.0000

Data Index: 606859 (C4 (Web Text))

Max Activating Token Index: 163

Text #4

Max Range: 4.2941. Min Range: -4.2941

Max Act: 3.7802. Min Act: -0.0000

Data Index: 1037416 (C4 (Web Text))

Max Activating Token Index: 102

Text #5

Max Range: 4.2941. Min Range: -4.2941

Max Act: 3.7896. Min Act: -0.0000

Data Index: 1074402 (C4 (Web Text))

Max Activating Token Index: 541

Text #6

Max Range: 4.2941. Min Range: -4.2941

Max Act: 3.3312. Min Act: -0.0000

Data Index: 937750 (C4 (Web Text))

Max Activating Token Index: 248

Text #7

Max Range: 4.2941. Min Range: -4.2941

Max Act: 3.4606. Min Act: -0.0000

Data Index: 380094 (C4 (Web Text))

Max Activating Token Index: 278

Text #8

Max Range: 4.2941. Min Range: -4.2941

Max Act: 3.1818. Min Act: -0.0000

Data Index: 146626 (C4 (Web Text))

Max Activating Token Index: 578

Text #9

Max Range: 4.2941. Min Range: -4.2941

Max Act: 3.4857. Min Act: -0.0000

Data Index: 1162499 (C4 (Web Text))

Max Activating Token Index: 552

Text #10

Max Range: 4.2941. Min Range: -4.2941

Max Act: 3.2726. Min Act: -0.0000

Data Index: 467420 (C4 (Web Text))

Max Activating Token Index: 672

Text #11

Max Range: 4.2941. Min Range: -4.2941

Max Act: 3.2414. Min Act: -0.0000

Data Index: 282202 (C4 (Web Text))

Max Activating Token Index: 265

Text #12

Max Range: 4.2941. Min Range: -4.2941

Max Act: 3.1534. Min Act: -0.0000

Data Index: 792042 (C4 (Web Text))

Max Activating Token Index: 157

Text #13

Max Range: 4.2941. Min Range: -4.2941

Max Act: 3.0498. Min Act: -0.0000

Data Index: 340228 (C4 (Web Text))

Max Activating Token Index: 640

Text #14

Max Range: 4.2941. Min Range: -4.2941

Max Act: 3.0400. Min Act: -0.0000

Data Index: 2059 (C4 (Web Text))

Max Activating Token Index: 927

Text #15

Max Range: 4.2941. Min Range: -4.2941

Max Act: 3.0088. Min Act: -0.0000

Data Index: 262234 (C4 (Web Text))

Max Activating Token Index: 906

Text #16

Max Range: 4.2941. Min Range: -4.2941

Max Act: 3.0099. Min Act: -0.0000

Data Index: 925592 (C4 (Web Text))

Max Activating Token Index: 53

Text #17

Max Range: 4.2941. Min Range: -4.2941

Max Act: 2.9861. Min Act: -0.0000

Data Index: 857250 (C4 (Web Text))

Max Activating Token Index: 700

Text #18

Max Range: 4.2941. Min Range: -4.2941

Max Act: 3.0770. Min Act: -0.0000

Data Index: 1164939 (C4 (Web Text))

Max Activating Token Index: 202

Text #19

Max Range: 4.2941. Min Range: -4.2941

Max Act: 2.8775. Min Act: -0.0000

Data Index: 908107 (C4 (Web Text))

Max Activating Token Index: 88