Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 5414 in Layer 10

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 0.8933. Min Range: -0.8933

Max Act: 0.8933. Min Act: -0.0000

Data Index: 1123473 (C4 (Web Text))

Max Activating Token Index: 599

Text #1

Max Range: 0.8933. Min Range: -0.8933

Max Act: 0.7693. Min Act: 0.0001

Data Index: 270061 (C4 (Web Text))

Max Activating Token Index: 232

Text #2

Max Range: 0.8933. Min Range: -0.8933

Max Act: 0.6635. Min Act: 0.0000

Data Index: 1298436 (C4 (Web Text))

Max Activating Token Index: 216

Text #3

Max Range: 0.8933. Min Range: -0.8933

Max Act: 0.6675. Min Act: -0.0000

Data Index: 556803 (C4 (Web Text))

Max Activating Token Index: 260

Text #4

Max Range: 0.8933. Min Range: -0.8933

Max Act: 0.6638. Min Act: -0.0000

Data Index: 1141906 (C4 (Web Text))

Max Activating Token Index: 21

Text #5

Max Range: 0.8933. Min Range: -0.8933

Max Act: 0.6152. Min Act: -0.0000

Data Index: 537084 (C4 (Web Text))

Max Activating Token Index: 202

Text #6

Max Range: 0.8933. Min Range: -0.8933

Max Act: 0.6080. Min Act: -0.0000

Data Index: 456082 (C4 (Web Text))

Max Activating Token Index: 558

Text #7

Max Range: 0.8933. Min Range: -0.8933

Max Act: 0.6051. Min Act: -0.0000

Data Index: 863257 (C4 (Web Text))

Max Activating Token Index: 69

Text #8

Max Range: 0.8933. Min Range: -0.8933

Max Act: 0.5719. Min Act: 0.0000

Data Index: 572162 (C4 (Web Text))

Max Activating Token Index: 89

Text #9

Max Range: 0.8933. Min Range: -0.8933

Max Act: 0.5933. Min Act: -0.0000

Data Index: 863359 (C4 (Web Text))

Max Activating Token Index: 228

Text #10

Max Range: 0.8933. Min Range: -0.8933

Max Act: 0.5634. Min Act: -0.0000

Data Index: 1064259 (C4 (Web Text))

Max Activating Token Index: 863

Text #11

Max Range: 0.8933. Min Range: -0.8933

Max Act: 0.5620. Min Act: -0.0000

Data Index: 1221248 (C4 (Web Text))

Max Activating Token Index: 828

Text #12

Max Range: 0.8933. Min Range: -0.8933

Max Act: 0.5590. Min Act: -0.0000

Data Index: 242319 (C4 (Web Text))

Max Activating Token Index: 135

Text #13

Max Range: 0.8933. Min Range: -0.8933

Max Act: 0.5714. Min Act: -0.0000

Data Index: 324978 (C4 (Web Text))

Max Activating Token Index: 918

Text #14

Max Range: 0.8933. Min Range: -0.8933

Max Act: 0.5864. Min Act: -0.0000

Data Index: 1056016 (C4 (Web Text))

Max Activating Token Index: 186

Text #15

Max Range: 0.8933. Min Range: -0.8933

Max Act: 0.5445. Min Act: -0.0000

Data Index: 290406 (C4 (Web Text))

Max Activating Token Index: 821

Text #16

Max Range: 0.8933. Min Range: -0.8933

Max Act: 0.5482. Min Act: 0.0001

Data Index: 1097890 (C4 (Web Text))

Max Activating Token Index: 643

Text #17

Max Range: 0.8933. Min Range: -0.8933

Max Act: 0.5315. Min Act: -0.0000

Data Index: 1332902 (C4 (Web Text))

Max Activating Token Index: 609

Text #18

Max Range: 0.8933. Min Range: -0.8933

Max Act: 0.5473. Min Act: -0.0000

Data Index: 1162388 (C4 (Web Text))

Max Activating Token Index: 189

Text #19

Max Range: 0.8933. Min Range: -0.8933

Max Act: 0.5292. Min Act: -0.0000

Data Index: 439401 (C4 (Web Text))

Max Activating Token Index: 800