Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3233 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.3393. Min Range: -0.3393

Max Act: 0.3015. Min Act: -0.0000

Data Index: 1034950 (C4 (Web Text))

Max Activating Token Index: 235

Text #1

Max Range: 0.3393. Min Range: -0.3393

Max Act: 0.3393. Min Act: -0.0000

Data Index: 435689 (C4 (Web Text))

Max Activating Token Index: 749

Text #2

Max Range: 0.3393. Min Range: -0.3393

Max Act: 0.3195. Min Act: -0.0000

Data Index: 1322378 (C4 (Web Text))

Max Activating Token Index: 1016

Text #3

Max Range: 0.3393. Min Range: -0.3393

Max Act: 0.3279. Min Act: -0.0000

Data Index: 1172594 (C4 (Web Text))

Max Activating Token Index: 603

Text #4

Max Range: 0.3393. Min Range: -0.3393

Max Act: 0.3179. Min Act: -0.0000

Data Index: 193323 (C4 (Web Text))

Max Activating Token Index: 783

Text #5

Max Range: 0.3393. Min Range: -0.3393

Max Act: 0.3374. Min Act: -0.0000

Data Index: 532890 (C4 (Web Text))

Max Activating Token Index: 937

Text #6

Max Range: 0.3393. Min Range: -0.3393

Max Act: 0.3297. Min Act: -0.0000

Data Index: 413071 (C4 (Web Text))

Max Activating Token Index: 775

Text #7

Max Range: 0.3393. Min Range: -0.3393

Max Act: 0.2938. Min Act: -0.0000

Data Index: 143567 (C4 (Web Text))

Max Activating Token Index: 556

Text #8

Max Range: 0.3393. Min Range: -0.3393

Max Act: 0.2910. Min Act: -0.0000

Data Index: 650621 (C4 (Web Text))

Max Activating Token Index: 903

Text #9

Max Range: 0.3393. Min Range: -0.3393

Max Act: 0.2828. Min Act: -0.0000

Data Index: 1479978 (Python Code)

Max Activating Token Index: 390

Text #10

Max Range: 0.3393. Min Range: -0.3393

Max Act: 0.2661. Min Act: -0.0000

Data Index: 624528 (C4 (Web Text))

Max Activating Token Index: 176

Text #11

Max Range: 0.3393. Min Range: -0.3393

Max Act: 0.2866. Min Act: -0.0000

Data Index: 473326 (C4 (Web Text))

Max Activating Token Index: 968

Text #12

Max Range: 0.3393. Min Range: -0.3393

Max Act: 0.3066. Min Act: -0.0000

Data Index: 945644 (C4 (Web Text))

Max Activating Token Index: 661

Text #13

Max Range: 0.3393. Min Range: -0.3393

Max Act: 0.2886. Min Act: -0.0000

Data Index: 122013 (C4 (Web Text))

Max Activating Token Index: 1000

Text #14

Max Range: 0.3393. Min Range: -0.3393

Max Act: 0.3001. Min Act: -0.0000

Data Index: 219752 (C4 (Web Text))

Max Activating Token Index: 366

Text #15

Max Range: 0.3393. Min Range: -0.3393

Max Act: 0.2855. Min Act: -0.0000

Data Index: 1091594 (C4 (Web Text))

Max Activating Token Index: 874

Text #16

Max Range: 0.3393. Min Range: -0.3393

Max Act: 0.2845. Min Act: -0.0000

Data Index: 143403 (C4 (Web Text))

Max Activating Token Index: 627

Text #17

Max Range: 0.3393. Min Range: -0.3393

Max Act: 0.2596. Min Act: -0.0000

Data Index: 1361838 (Python Code)

Max Activating Token Index: 335

Text #18

Max Range: 0.3393. Min Range: -0.3393

Max Act: 0.2649. Min Act: -0.0000

Data Index: 1311952 (C4 (Web Text))

Max Activating Token Index: 621

Text #19

Max Range: 0.3393. Min Range: -0.3393

Max Act: 0.2724. Min Act: -0.0000

Data Index: 1312827 (C4 (Web Text))

Max Activating Token Index: 209