Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3904 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.6283. Min Range: -1.6283

Max Act: 1.6283. Min Act: -0.0000

Data Index: 755133 (C4 (Web Text))

Max Activating Token Index: 661

Text #1

Max Range: 1.6283. Min Range: -1.6283

Max Act: 1.3961. Min Act: -0.0000

Data Index: 1180003 (C4 (Web Text))

Max Activating Token Index: 761

Text #2

Max Range: 1.6283. Min Range: -1.6283

Max Act: 1.2337. Min Act: -0.0000

Data Index: 235464 (C4 (Web Text))

Max Activating Token Index: 698

Text #3

Max Range: 1.6283. Min Range: -1.6283

Max Act: 1.1844. Min Act: -0.0000

Data Index: 627095 (C4 (Web Text))

Max Activating Token Index: 666

Text #4

Max Range: 1.6283. Min Range: -1.6283

Max Act: 1.1026. Min Act: -0.0000

Data Index: 1286999 (C4 (Web Text))

Max Activating Token Index: 232

Text #5

Max Range: 1.6283. Min Range: -1.6283

Max Act: 1.1171. Min Act: -0.0000

Data Index: 93186 (C4 (Web Text))

Max Activating Token Index: 411

Text #6

Max Range: 1.6283. Min Range: -1.6283

Max Act: 1.0437. Min Act: -0.0000

Data Index: 843750 (C4 (Web Text))

Max Activating Token Index: 392

Text #7

Max Range: 1.6283. Min Range: -1.6283

Max Act: 1.0681. Min Act: -0.0000

Data Index: 318915 (C4 (Web Text))

Max Activating Token Index: 239

Text #8

Max Range: 1.6283. Min Range: -1.6283

Max Act: 0.9168. Min Act: -0.0000

Data Index: 636001 (C4 (Web Text))

Max Activating Token Index: 888

Text #9

Max Range: 1.6283. Min Range: -1.6283

Max Act: 0.9899. Min Act: -0.0000

Data Index: 535707 (C4 (Web Text))

Max Activating Token Index: 78

Text #10

Max Range: 1.6283. Min Range: -1.6283

Max Act: 0.9654. Min Act: -0.0000

Data Index: 802501 (C4 (Web Text))

Max Activating Token Index: 803

Text #11

Max Range: 1.6283. Min Range: -1.6283

Max Act: 0.9494. Min Act: -0.0000

Data Index: 901997 (C4 (Web Text))

Max Activating Token Index: 16

Text #12

Max Range: 1.6283. Min Range: -1.6283

Max Act: 0.9493. Min Act: -0.0000

Data Index: 458069 (C4 (Web Text))

Max Activating Token Index: 720

Text #13

Max Range: 1.6283. Min Range: -1.6283

Max Act: 0.9387. Min Act: -0.0000

Data Index: 1047349 (C4 (Web Text))

Max Activating Token Index: 718

Text #14

Max Range: 1.6283. Min Range: -1.6283

Max Act: 0.9266. Min Act: -0.0000

Data Index: 529013 (C4 (Web Text))

Max Activating Token Index: 507

Text #15

Max Range: 1.6283. Min Range: -1.6283

Max Act: 0.9044. Min Act: -0.0000

Data Index: 17409 (C4 (Web Text))

Max Activating Token Index: 813

Text #16

Max Range: 1.6283. Min Range: -1.6283

Max Act: 0.9807. Min Act: -0.0000

Data Index: 25641 (C4 (Web Text))

Max Activating Token Index: 973

Text #17

Max Range: 1.6283. Min Range: -1.6283

Max Act: 0.9428. Min Act: -0.0000

Data Index: 677932 (C4 (Web Text))

Max Activating Token Index: 730

Text #18

Max Range: 1.6283. Min Range: -1.6283

Max Act: 0.9341. Min Act: -0.0000

Data Index: 355726 (C4 (Web Text))

Max Activating Token Index: 177

Text #19

Max Range: 1.6283. Min Range: -1.6283

Max Act: 0.9082. Min Act: -0.0000

Data Index: 289518 (C4 (Web Text))

Max Activating Token Index: 672