Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1326 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.2057. Min Range: -1.2057

Max Act: 1.2057. Min Act: -0.0000

Data Index: 11965 (C4 (Web Text))

Max Activating Token Index: 959

Text #1

Max Range: 1.2057. Min Range: -1.2057

Max Act: 1.0290. Min Act: -0.0000

Data Index: 1318791 (C4 (Web Text))

Max Activating Token Index: 814

Text #2

Max Range: 1.2057. Min Range: -1.2057

Max Act: 0.9196. Min Act: -0.0000

Data Index: 1287471 (C4 (Web Text))

Max Activating Token Index: 389

Text #3

Max Range: 1.2057. Min Range: -1.2057

Max Act: 0.8758. Min Act: -0.0000

Data Index: 439425 (C4 (Web Text))

Max Activating Token Index: 133

Text #4

Max Range: 1.2057. Min Range: -1.2057

Max Act: 0.8708. Min Act: -0.0000

Data Index: 258484 (C4 (Web Text))

Max Activating Token Index: 723

Text #5

Max Range: 1.2057. Min Range: -1.2057

Max Act: 0.8624. Min Act: -0.0000

Data Index: 377074 (C4 (Web Text))

Max Activating Token Index: 811

Text #6

Max Range: 1.2057. Min Range: -1.2057

Max Act: 0.8294. Min Act: -0.0000

Data Index: 298674 (C4 (Web Text))

Max Activating Token Index: 863

Text #7

Max Range: 1.2057. Min Range: -1.2057

Max Act: 0.7398. Min Act: -0.0000

Data Index: 161289 (C4 (Web Text))

Max Activating Token Index: 852

Text #8

Max Range: 1.2057. Min Range: -1.2057

Max Act: 0.9328. Min Act: -0.0000

Data Index: 1328687 (C4 (Web Text))

Max Activating Token Index: 299

Text #9

Max Range: 1.2057. Min Range: -1.2057

Max Act: 0.8110. Min Act: -0.0000

Data Index: 357180 (C4 (Web Text))

Max Activating Token Index: 582

Text #10

Max Range: 1.2057. Min Range: -1.2057

Max Act: 0.8269. Min Act: -0.0000

Data Index: 6808 (C4 (Web Text))

Max Activating Token Index: 701

Text #11

Max Range: 1.2057. Min Range: -1.2057

Max Act: 0.8022. Min Act: -0.0000

Data Index: 1282235 (C4 (Web Text))

Max Activating Token Index: 712

Text #12

Max Range: 1.2057. Min Range: -1.2057

Max Act: 0.8218. Min Act: -0.0000

Data Index: 205078 (C4 (Web Text))

Max Activating Token Index: 80

Text #13

Max Range: 1.2057. Min Range: -1.2057

Max Act: 0.7583. Min Act: -0.0000

Data Index: 18583 (C4 (Web Text))

Max Activating Token Index: 201

Text #14

Max Range: 1.2057. Min Range: -1.2057

Max Act: 0.8055. Min Act: -0.0000

Data Index: 1154942 (C4 (Web Text))

Max Activating Token Index: 175

Text #15

Max Range: 1.2057. Min Range: -1.2057

Max Act: 0.7553. Min Act: -0.0000

Data Index: 27832 (C4 (Web Text))

Max Activating Token Index: 1012

Text #16

Max Range: 1.2057. Min Range: -1.2057

Max Act: 0.7664. Min Act: -0.0000

Data Index: 158299 (C4 (Web Text))

Max Activating Token Index: 500

Text #17

Max Range: 1.2057. Min Range: -1.2057

Max Act: 0.7643. Min Act: -0.0000

Data Index: 883170 (C4 (Web Text))

Max Activating Token Index: 337

Text #18

Max Range: 1.2057. Min Range: -1.2057

Max Act: 0.7573. Min Act: -0.0000

Data Index: 69492 (C4 (Web Text))

Max Activating Token Index: 625

Text #19

Max Range: 1.2057. Min Range: -1.2057

Max Act: 0.7437. Min Act: -0.0000

Data Index: 1348182 (C4 (Web Text))

Max Activating Token Index: 406