Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4199 in Layer 11

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.1168. Min Range: -1.1168

Max Act: 1.1168. Min Act: -0.0000

Data Index: 1183107 (C4 (Web Text))

Max Activating Token Index: 177

Text #1

Max Range: 1.1168. Min Range: -1.1168

Max Act: 1.0447. Min Act: -0.0000

Data Index: 819351 (C4 (Web Text))

Max Activating Token Index: 926

Text #2

Max Range: 1.1168. Min Range: -1.1168

Max Act: 0.9648. Min Act: -0.0000

Data Index: 520873 (C4 (Web Text))

Max Activating Token Index: 32

Text #3

Max Range: 1.1168. Min Range: -1.1168

Max Act: 0.8495. Min Act: -0.0000

Data Index: 1292642 (C4 (Web Text))

Max Activating Token Index: 972

Text #4

Max Range: 1.1168. Min Range: -1.1168

Max Act: 0.8022. Min Act: 0.0000

Data Index: 707627 (C4 (Web Text))

Max Activating Token Index: 96

Text #5

Max Range: 1.1168. Min Range: -1.1168

Max Act: 0.7549. Min Act: 0.0000

Data Index: 201735 (C4 (Web Text))

Max Activating Token Index: 797

Text #6

Max Range: 1.1168. Min Range: -1.1168

Max Act: 0.7667. Min Act: -0.0000

Data Index: 1185496 (C4 (Web Text))

Max Activating Token Index: 287

Text #7

Max Range: 1.1168. Min Range: -1.1168

Max Act: 0.7941. Min Act: -0.0000

Data Index: 250334 (C4 (Web Text))

Max Activating Token Index: 46

Text #8

Max Range: 1.1168. Min Range: -1.1168

Max Act: 0.7431. Min Act: 0.0000

Data Index: 1192269 (C4 (Web Text))

Max Activating Token Index: 8

Text #9

Max Range: 1.1168. Min Range: -1.1168

Max Act: 0.7877. Min Act: -0.0000

Data Index: 776620 (C4 (Web Text))

Max Activating Token Index: 931

Text #10

Max Range: 1.1168. Min Range: -1.1168

Max Act: 0.7570. Min Act: -0.0000

Data Index: 517714 (C4 (Web Text))

Max Activating Token Index: 139

Text #11

Max Range: 1.1168. Min Range: -1.1168

Max Act: 0.7554. Min Act: 0.0000

Data Index: 103676 (C4 (Web Text))

Max Activating Token Index: 94

Text #12

Max Range: 1.1168. Min Range: -1.1168

Max Act: 0.7383. Min Act: -0.0000

Data Index: 434331 (C4 (Web Text))

Max Activating Token Index: 529

Text #13

Max Range: 1.1168. Min Range: -1.1168

Max Act: 0.7130. Min Act: 0.0000

Data Index: 1165856 (C4 (Web Text))

Max Activating Token Index: 763

Text #14

Max Range: 1.1168. Min Range: -1.1168

Max Act: 0.7245. Min Act: 0.0000

Data Index: 1279669 (C4 (Web Text))

Max Activating Token Index: 946

Text #15

Max Range: 1.1168. Min Range: -1.1168

Max Act: 0.7158. Min Act: -0.0000

Data Index: 1185331 (C4 (Web Text))

Max Activating Token Index: 951

Text #16

Max Range: 1.1168. Min Range: -1.1168

Max Act: 0.7386. Min Act: -0.0000

Data Index: 778415 (C4 (Web Text))

Max Activating Token Index: 626

Text #17

Max Range: 1.1168. Min Range: -1.1168

Max Act: 0.7204. Min Act: -0.0000

Data Index: 1140761 (C4 (Web Text))

Max Activating Token Index: 479

Text #18

Max Range: 1.1168. Min Range: -1.1168

Max Act: 0.7165. Min Act: 0.0000

Data Index: 117187 (C4 (Web Text))

Max Activating Token Index: 893

Text #19

Max Range: 1.1168. Min Range: -1.1168

Max Act: 0.6994. Min Act: -0.0000

Data Index: 1080387 (C4 (Web Text))

Max Activating Token Index: 839