Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 432 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 2.8251. Min Range: -2.8251

Max Act: 2.8251. Min Act: -0.0000

Data Index: 1478077 (Python Code)

Max Activating Token Index: 260

Text #1

Max Range: 2.8251. Min Range: -2.8251

Max Act: 2.6913. Min Act: -0.0000

Data Index: 1523219 (Python Code)

Max Activating Token Index: 160

Text #2

Max Range: 2.8251. Min Range: -2.8251

Max Act: 2.7716. Min Act: -0.0000

Data Index: 1606743 (Python Code)

Max Activating Token Index: 926

Text #3

Max Range: 2.8251. Min Range: -2.8251

Max Act: 2.4719. Min Act: -0.0000

Data Index: 1534747 (Python Code)

Max Activating Token Index: 828

Text #4

Max Range: 2.8251. Min Range: -2.8251

Max Act: 2.4786. Min Act: -0.0000

Data Index: 1461060 (Python Code)

Max Activating Token Index: 897

Text #5

Max Range: 2.8251. Min Range: -2.8251

Max Act: 2.3366. Min Act: -0.0000

Data Index: 1398263 (Python Code)

Max Activating Token Index: 183

Text #6

Max Range: 2.8251. Min Range: -2.8251

Max Act: 2.2205. Min Act: -0.0000

Data Index: 1393309 (Python Code)

Max Activating Token Index: 609

Text #7

Max Range: 2.8251. Min Range: -2.8251

Max Act: 2.3302. Min Act: -0.0000

Data Index: 1549125 (Python Code)

Max Activating Token Index: 755

Text #8

Max Range: 2.8251. Min Range: -2.8251

Max Act: 2.2902. Min Act: -0.0000

Data Index: 1369883 (Python Code)

Max Activating Token Index: 1007

Text #9

Max Range: 2.8251. Min Range: -2.8251

Max Act: 2.4300. Min Act: -0.0000

Data Index: 1635978 (Python Code)

Max Activating Token Index: 809

Text #10

Max Range: 2.8251. Min Range: -2.8251

Max Act: 2.4926. Min Act: -0.0000

Data Index: 1630064 (Python Code)

Max Activating Token Index: 821

Text #11

Max Range: 2.8251. Min Range: -2.8251

Max Act: 2.3929. Min Act: -0.0000

Data Index: 1604860 (Python Code)

Max Activating Token Index: 482

Text #12

Max Range: 2.8251. Min Range: -2.8251

Max Act: 2.2724. Min Act: -0.0000

Data Index: 1585784 (Python Code)

Max Activating Token Index: 833

Text #13

Max Range: 2.8251. Min Range: -2.8251

Max Act: 2.5076. Min Act: -0.0000

Data Index: 1462170 (Python Code)

Max Activating Token Index: 195

Text #14

Max Range: 2.8251. Min Range: -2.8251

Max Act: 2.3699. Min Act: -0.0000

Data Index: 1526281 (Python Code)

Max Activating Token Index: 270

Text #15

Max Range: 2.8251. Min Range: -2.8251

Max Act: 2.1574. Min Act: -0.0000

Data Index: 1616042 (Python Code)

Max Activating Token Index: 1010

Text #16

Max Range: 2.8251. Min Range: -2.8251

Max Act: 2.3454. Min Act: -0.0000

Data Index: 1445064 (Python Code)

Max Activating Token Index: 56

Text #17

Max Range: 2.8251. Min Range: -2.8251

Max Act: 2.1057. Min Act: -0.0000

Data Index: 1407724 (Python Code)

Max Activating Token Index: 704

Text #18

Max Range: 2.8251. Min Range: -2.8251

Max Act: 2.3400. Min Act: -0.0000

Data Index: 1447310 (Python Code)

Max Activating Token Index: 210

Text #19

Max Range: 2.8251. Min Range: -2.8251

Max Act: 2.1975. Min Act: -0.0000

Data Index: 1457644 (Python Code)

Max Activating Token Index: 65