Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1157 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.1661. Min Range: -1.1661

Max Act: 1.1661. Min Act: -0.0000

Data Index: 1356013 (C4 (Web Text))

Max Activating Token Index: 763

Text #1

Max Range: 1.1661. Min Range: -1.1661

Max Act: 1.1021. Min Act: -0.0000

Data Index: 724578 (C4 (Web Text))

Max Activating Token Index: 314

Text #2

Max Range: 1.1661. Min Range: -1.1661

Max Act: 1.1017. Min Act: -0.0000

Data Index: 645282 (C4 (Web Text))

Max Activating Token Index: 1017

Text #3

Max Range: 1.1661. Min Range: -1.1661

Max Act: 1.0472. Min Act: -0.0000

Data Index: 1324825 (C4 (Web Text))

Max Activating Token Index: 715

Text #4

Max Range: 1.1661. Min Range: -1.1661

Max Act: 1.0398. Min Act: -0.0000

Data Index: 417458 (C4 (Web Text))

Max Activating Token Index: 32

Text #5

Max Range: 1.1661. Min Range: -1.1661

Max Act: 1.0889. Min Act: -0.0000

Data Index: 979580 (C4 (Web Text))

Max Activating Token Index: 286

Text #6

Max Range: 1.1661. Min Range: -1.1661

Max Act: 1.0266. Min Act: -0.0000

Data Index: 240787 (C4 (Web Text))

Max Activating Token Index: 398

Text #7

Max Range: 1.1661. Min Range: -1.1661

Max Act: 1.0042. Min Act: -0.0000

Data Index: 1086574 (C4 (Web Text))

Max Activating Token Index: 1016

Text #8

Max Range: 1.1661. Min Range: -1.1661

Max Act: 1.0477. Min Act: -0.0000

Data Index: 1347704 (C4 (Web Text))

Max Activating Token Index: 1020

Text #9

Max Range: 1.1661. Min Range: -1.1661

Max Act: 1.0111. Min Act: -0.0000

Data Index: 46616 (C4 (Web Text))

Max Activating Token Index: 240

Text #10

Max Range: 1.1661. Min Range: -1.1661

Max Act: 0.8860. Min Act: -0.0000

Data Index: 141061 (C4 (Web Text))

Max Activating Token Index: 427

Text #11

Max Range: 1.1661. Min Range: -1.1661

Max Act: 0.9335. Min Act: -0.0000

Data Index: 489492 (C4 (Web Text))

Max Activating Token Index: 308

Text #12

Max Range: 1.1661. Min Range: -1.1661

Max Act: 0.9190. Min Act: -0.0000

Data Index: 102300 (C4 (Web Text))

Max Activating Token Index: 570

Text #13

Max Range: 1.1661. Min Range: -1.1661

Max Act: 0.9098. Min Act: -0.0000

Data Index: 78463 (C4 (Web Text))

Max Activating Token Index: 65

Text #14

Max Range: 1.1661. Min Range: -1.1661

Max Act: 0.9638. Min Act: -0.0000

Data Index: 1337921 (C4 (Web Text))

Max Activating Token Index: 237

Text #15

Max Range: 1.1661. Min Range: -1.1661

Max Act: 0.8435. Min Act: -0.0000

Data Index: 454941 (C4 (Web Text))

Max Activating Token Index: 999

Text #16

Max Range: 1.1661. Min Range: -1.1661

Max Act: 0.9038. Min Act: -0.0000

Data Index: 346012 (C4 (Web Text))

Max Activating Token Index: 559

Text #17

Max Range: 1.1661. Min Range: -1.1661

Max Act: 0.8948. Min Act: -0.0000

Data Index: 1217137 (C4 (Web Text))

Max Activating Token Index: 698

Text #18

Max Range: 1.1661. Min Range: -1.1661

Max Act: 0.8857. Min Act: -0.0000

Data Index: 798417 (C4 (Web Text))

Max Activating Token Index: 219

Text #19

Max Range: 1.1661. Min Range: -1.1661

Max Act: 0.8836. Min Act: -0.0000

Data Index: 798149 (C4 (Web Text))

Max Activating Token Index: 430