Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3728 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 2.7773. Min Range: -2.7773

Max Act: 2.7773. Min Act: -0.0000

Data Index: 789565 (C4 (Web Text))

Max Activating Token Index: 176

Text #1

Max Range: 2.7773. Min Range: -2.7773

Max Act: 2.4277. Min Act: -0.0000

Data Index: 367020 (C4 (Web Text))

Max Activating Token Index: 88

Text #2

Max Range: 2.7773. Min Range: -2.7773

Max Act: 2.5720. Min Act: -0.0000

Data Index: 564993 (C4 (Web Text))

Max Activating Token Index: 12

Text #3

Max Range: 2.7773. Min Range: -2.7773

Max Act: 2.5337. Min Act: -0.0000

Data Index: 614016 (C4 (Web Text))

Max Activating Token Index: 568

Text #4

Max Range: 2.7773. Min Range: -2.7773

Max Act: 2.4687. Min Act: -0.0000

Data Index: 1305402 (C4 (Web Text))

Max Activating Token Index: 679

Text #5

Max Range: 2.7773. Min Range: -2.7773

Max Act: 2.3447. Min Act: -0.0000

Data Index: 153426 (C4 (Web Text))

Max Activating Token Index: 531

Text #6

Max Range: 2.7773. Min Range: -2.7773

Max Act: 2.2858. Min Act: -0.0000

Data Index: 37765 (C4 (Web Text))

Max Activating Token Index: 365

Text #7

Max Range: 2.7773. Min Range: -2.7773

Max Act: 2.3440. Min Act: -0.0000

Data Index: 987494 (C4 (Web Text))

Max Activating Token Index: 425

Text #8

Max Range: 2.7773. Min Range: -2.7773

Max Act: 2.3173. Min Act: -0.0000

Data Index: 381228 (C4 (Web Text))

Max Activating Token Index: 928

Text #9

Max Range: 2.7773. Min Range: -2.7773

Max Act: 2.2095. Min Act: -0.0000

Data Index: 519790 (C4 (Web Text))

Max Activating Token Index: 202

Text #10

Max Range: 2.7773. Min Range: -2.7773

Max Act: 2.1890. Min Act: -0.0000

Data Index: 341730 (C4 (Web Text))

Max Activating Token Index: 820

Text #11

Max Range: 2.7773. Min Range: -2.7773

Max Act: 2.0415. Min Act: 0.0000

Data Index: 1142119 (C4 (Web Text))

Max Activating Token Index: 868

Text #12

Max Range: 2.7773. Min Range: -2.7773

Max Act: 2.2694. Min Act: -0.0000

Data Index: 222438 (C4 (Web Text))

Max Activating Token Index: 986

Text #13

Max Range: 2.7773. Min Range: -2.7773

Max Act: 2.2618. Min Act: -0.0000

Data Index: 1039941 (C4 (Web Text))

Max Activating Token Index: 15

Text #14

Max Range: 2.7773. Min Range: -2.7773

Max Act: 2.2495. Min Act: -0.0000

Data Index: 112849 (C4 (Web Text))

Max Activating Token Index: 106

Text #15

Max Range: 2.7773. Min Range: -2.7773

Max Act: 2.0860. Min Act: -0.0000

Data Index: 1408828 (Python Code)

Max Activating Token Index: 979

Text #16

Max Range: 2.7773. Min Range: -2.7773

Max Act: 2.1937. Min Act: -0.0000

Data Index: 947062 (C4 (Web Text))

Max Activating Token Index: 396

Text #17

Max Range: 2.7773. Min Range: -2.7773

Max Act: 2.0793. Min Act: -0.0000

Data Index: 593225 (C4 (Web Text))

Max Activating Token Index: 10

Text #18

Max Range: 2.7773. Min Range: -2.7773

Max Act: 2.1726. Min Act: -0.0000

Data Index: 1054315 (C4 (Web Text))

Max Activating Token Index: 967

Text #19

Max Range: 2.7773. Min Range: -2.7773

Max Act: 2.0564. Min Act: -0.0000

Data Index: 76806 (C4 (Web Text))

Max Activating Token Index: 246