Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 5491 in Layer 11

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.2072. Min Range: -1.2072

Max Act: 1.2072. Min Act: -0.0000

Data Index: 630826 (C4 (Web Text))

Max Activating Token Index: 994

Text #1

Max Range: 1.2072. Min Range: -1.2072

Max Act: 0.9412. Min Act: -0.0000

Data Index: 857511 (C4 (Web Text))

Max Activating Token Index: 692

Text #2

Max Range: 1.2072. Min Range: -1.2072

Max Act: 0.8757. Min Act: -0.0000

Data Index: 972291 (C4 (Web Text))

Max Activating Token Index: 450

Text #3

Max Range: 1.2072. Min Range: -1.2072

Max Act: 0.8367. Min Act: -0.0000

Data Index: 1038350 (C4 (Web Text))

Max Activating Token Index: 838

Text #4

Max Range: 1.2072. Min Range: -1.2072

Max Act: 0.6970. Min Act: -0.0000

Data Index: 509073 (C4 (Web Text))

Max Activating Token Index: 624

Text #5

Max Range: 1.2072. Min Range: -1.2072

Max Act: 0.8426. Min Act: -0.0000

Data Index: 941652 (C4 (Web Text))

Max Activating Token Index: 865

Text #6

Max Range: 1.2072. Min Range: -1.2072

Max Act: 0.7029. Min Act: -0.0000

Data Index: 1298595 (C4 (Web Text))

Max Activating Token Index: 254

Text #7

Max Range: 1.2072. Min Range: -1.2072

Max Act: 0.7027. Min Act: -0.0000

Data Index: 469589 (C4 (Web Text))

Max Activating Token Index: 975

Text #8

Max Range: 1.2072. Min Range: -1.2072

Max Act: 0.6914. Min Act: -0.0000

Data Index: 342202 (C4 (Web Text))

Max Activating Token Index: 865

Text #9

Max Range: 1.2072. Min Range: -1.2072

Max Act: 0.6812. Min Act: -0.0000

Data Index: 436738 (C4 (Web Text))

Max Activating Token Index: 1013

Text #10

Max Range: 1.2072. Min Range: -1.2072

Max Act: 0.6398. Min Act: -0.0000

Data Index: 105164 (C4 (Web Text))

Max Activating Token Index: 1006

Text #11

Max Range: 1.2072. Min Range: -1.2072

Max Act: 0.6790. Min Act: -0.0000

Data Index: 1328398 (C4 (Web Text))

Max Activating Token Index: 113

Text #12

Max Range: 1.2072. Min Range: -1.2072

Max Act: 0.5919. Min Act: -0.0000

Data Index: 65715 (C4 (Web Text))

Max Activating Token Index: 261

Text #13

Max Range: 1.2072. Min Range: -1.2072

Max Act: 0.6072. Min Act: -0.0000

Data Index: 173485 (C4 (Web Text))

Max Activating Token Index: 711

Text #14

Max Range: 1.2072. Min Range: -1.2072

Max Act: 0.6858. Min Act: -0.0000

Data Index: 269856 (C4 (Web Text))

Max Activating Token Index: 858

Text #15

Max Range: 1.2072. Min Range: -1.2072

Max Act: 0.6410. Min Act: -0.0000

Data Index: 127987 (C4 (Web Text))

Max Activating Token Index: 1014

Text #16

Max Range: 1.2072. Min Range: -1.2072

Max Act: 0.6370. Min Act: -0.0000

Data Index: 208355 (C4 (Web Text))

Max Activating Token Index: 416

Text #17

Max Range: 1.2072. Min Range: -1.2072

Max Act: 0.6192. Min Act: -0.0000

Data Index: 288160 (C4 (Web Text))

Max Activating Token Index: 273

Text #18

Max Range: 1.2072. Min Range: -1.2072

Max Act: 0.6688. Min Act: -0.0000

Data Index: 1244051 (C4 (Web Text))

Max Activating Token Index: 833

Text #19

Max Range: 1.2072. Min Range: -1.2072

Max Act: 0.6501. Min Act: -0.0000

Data Index: 2900 (C4 (Web Text))

Max Activating Token Index: 183