Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3895 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 3.7722. Min Range: -3.7722

Max Act: 3.6632. Min Act: -0.0000

Data Index: 1335942 (C4 (Web Text))

Max Activating Token Index: 577

Text #1

Max Range: 3.7722. Min Range: -3.7722

Max Act: 3.7722. Min Act: -0.0000

Data Index: 368912 (C4 (Web Text))

Max Activating Token Index: 44

Text #2

Max Range: 3.7722. Min Range: -3.7722

Max Act: 3.0391. Min Act: -0.0000

Data Index: 55512 (C4 (Web Text))

Max Activating Token Index: 956

Text #3

Max Range: 3.7722. Min Range: -3.7722

Max Act: 3.5512. Min Act: -0.0000

Data Index: 1254817 (C4 (Web Text))

Max Activating Token Index: 761

Text #4

Max Range: 3.7722. Min Range: -3.7722

Max Act: 3.4068. Min Act: -0.0000

Data Index: 597397 (C4 (Web Text))

Max Activating Token Index: 301

Text #5

Max Range: 3.7722. Min Range: -3.7722

Max Act: 3.2148. Min Act: -0.0000

Data Index: 642748 (C4 (Web Text))

Max Activating Token Index: 161

Text #6

Max Range: 3.7722. Min Range: -3.7722

Max Act: 3.1568. Min Act: -0.0000

Data Index: 591967 (C4 (Web Text))

Max Activating Token Index: 502

Text #7

Max Range: 3.7722. Min Range: -3.7722

Max Act: 3.4631. Min Act: -0.0000

Data Index: 597376 (C4 (Web Text))

Max Activating Token Index: 313

Text #8

Max Range: 3.7722. Min Range: -3.7722

Max Act: 3.1858. Min Act: -0.0000

Data Index: 151556 (C4 (Web Text))

Max Activating Token Index: 142

Text #9

Max Range: 3.7722. Min Range: -3.7722

Max Act: 3.3357. Min Act: -0.0000

Data Index: 1171448 (C4 (Web Text))

Max Activating Token Index: 295

Text #10

Max Range: 3.7722. Min Range: -3.7722

Max Act: 2.7885. Min Act: -0.0000

Data Index: 199285 (C4 (Web Text))

Max Activating Token Index: 118

Text #11

Max Range: 3.7722. Min Range: -3.7722

Max Act: 3.0718. Min Act: -0.0000

Data Index: 1241556 (C4 (Web Text))

Max Activating Token Index: 302

Text #12

Max Range: 3.7722. Min Range: -3.7722

Max Act: 3.4931. Min Act: -0.0000

Data Index: 374707 (C4 (Web Text))

Max Activating Token Index: 254

Text #13

Max Range: 3.7722. Min Range: -3.7722

Max Act: 3.0427. Min Act: -0.0000

Data Index: 596961 (C4 (Web Text))

Max Activating Token Index: 392

Text #14

Max Range: 3.7722. Min Range: -3.7722

Max Act: 2.8437. Min Act: -0.0000

Data Index: 638661 (C4 (Web Text))

Max Activating Token Index: 51

Text #15

Max Range: 3.7722. Min Range: -3.7722

Max Act: 2.9573. Min Act: -0.0000

Data Index: 363290 (C4 (Web Text))

Max Activating Token Index: 662

Text #16

Max Range: 3.7722. Min Range: -3.7722

Max Act: 2.7171. Min Act: -0.0000

Data Index: 501176 (C4 (Web Text))

Max Activating Token Index: 389

Text #17

Max Range: 3.7722. Min Range: -3.7722

Max Act: 2.8107. Min Act: -0.0000

Data Index: 901235 (C4 (Web Text))

Max Activating Token Index: 807

Text #18

Max Range: 3.7722. Min Range: -3.7722

Max Act: 3.0855. Min Act: -0.0000

Data Index: 63800 (C4 (Web Text))

Max Activating Token Index: 628

Text #19

Max Range: 3.7722. Min Range: -3.7722

Max Act: 2.7941. Min Act: -0.0000

Data Index: 287123 (C4 (Web Text))

Max Activating Token Index: 379