Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 111 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 2.1751. Min Range: -2.1751

Max Act: 2.1751. Min Act: -0.0000

Data Index: 1356880 (C4 (Web Text))

Max Activating Token Index: 251

Text #1

Max Range: 2.1751. Min Range: -2.1751

Max Act: 2.1512. Min Act: -0.0000

Data Index: 456487 (C4 (Web Text))

Max Activating Token Index: 592

Text #2

Max Range: 2.1751. Min Range: -2.1751

Max Act: 1.8743. Min Act: -0.0000

Data Index: 1297360 (C4 (Web Text))

Max Activating Token Index: 831

Text #3

Max Range: 2.1751. Min Range: -2.1751

Max Act: 1.9654. Min Act: -0.0000

Data Index: 1371908 (Python Code)

Max Activating Token Index: 914

Text #4

Max Range: 2.1751. Min Range: -2.1751

Max Act: 1.9045. Min Act: -0.0000

Data Index: 301372 (C4 (Web Text))

Max Activating Token Index: 979

Text #5

Max Range: 2.1751. Min Range: -2.1751

Max Act: 1.8705. Min Act: -0.0000

Data Index: 126075 (C4 (Web Text))

Max Activating Token Index: 196

Text #6

Max Range: 2.1751. Min Range: -2.1751

Max Act: 1.9774. Min Act: -0.0000

Data Index: 729220 (C4 (Web Text))

Max Activating Token Index: 827

Text #7

Max Range: 2.1751. Min Range: -2.1751

Max Act: 1.8595. Min Act: -0.0000

Data Index: 833299 (C4 (Web Text))

Max Activating Token Index: 524

Text #8

Max Range: 2.1751. Min Range: -2.1751

Max Act: 1.7317. Min Act: -0.0000

Data Index: 182458 (C4 (Web Text))

Max Activating Token Index: 164

Text #9

Max Range: 2.1751. Min Range: -2.1751

Max Act: 1.9243. Min Act: -0.0000

Data Index: 38136 (C4 (Web Text))

Max Activating Token Index: 679

Text #10

Max Range: 2.1751. Min Range: -2.1751

Max Act: 1.8911. Min Act: -0.0000

Data Index: 1209015 (C4 (Web Text))

Max Activating Token Index: 227

Text #11

Max Range: 2.1751. Min Range: -2.1751

Max Act: 1.7584. Min Act: -0.0000

Data Index: 1308691 (C4 (Web Text))

Max Activating Token Index: 929

Text #12

Max Range: 2.1751. Min Range: -2.1751

Max Act: 1.6423. Min Act: -0.0000

Data Index: 725314 (C4 (Web Text))

Max Activating Token Index: 863

Text #13

Max Range: 2.1751. Min Range: -2.1751

Max Act: 1.9207. Min Act: -0.0000

Data Index: 1221452 (C4 (Web Text))

Max Activating Token Index: 733

Text #14

Max Range: 2.1751. Min Range: -2.1751

Max Act: 1.6296. Min Act: -0.0000

Data Index: 1277020 (C4 (Web Text))

Max Activating Token Index: 204

Text #15

Max Range: 2.1751. Min Range: -2.1751

Max Act: 1.6161. Min Act: -0.0000

Data Index: 1617636 (Python Code)

Max Activating Token Index: 216

Text #16

Max Range: 2.1751. Min Range: -2.1751

Max Act: 1.6718. Min Act: -0.0000

Data Index: 1442291 (Python Code)

Max Activating Token Index: 133

Text #17

Max Range: 2.1751. Min Range: -2.1751

Max Act: 1.6053. Min Act: -0.0000

Data Index: 1022448 (C4 (Web Text))

Max Activating Token Index: 146

Text #18

Max Range: 2.1751. Min Range: -2.1751

Max Act: 1.5969. Min Act: -0.0000

Data Index: 1373953 (Python Code)

Max Activating Token Index: 502

Text #19

Max Range: 2.1751. Min Range: -2.1751

Max Act: 1.5868. Min Act: -0.0000

Data Index: 972482 (C4 (Web Text))

Max Activating Token Index: 911