Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2067 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 3.4005. Min Range: -3.4005

Max Act: 3.4005. Min Act: -0.0000

Data Index: 1559384 (Python Code)

Max Activating Token Index: 262

Text #1

Max Range: 3.4005. Min Range: -3.4005

Max Act: 2.9899. Min Act: -0.0000

Data Index: 1570806 (Python Code)

Max Activating Token Index: 62

Text #2

Max Range: 3.4005. Min Range: -3.4005

Max Act: 3.1076. Min Act: -0.0000

Data Index: 1546636 (Python Code)

Max Activating Token Index: 107

Text #3

Max Range: 3.4005. Min Range: -3.4005

Max Act: 2.8930. Min Act: -0.0000

Data Index: 1497493 (Python Code)

Max Activating Token Index: 232

Text #4

Max Range: 3.4005. Min Range: -3.4005

Max Act: 2.9833. Min Act: -0.0000

Data Index: 1462383 (Python Code)

Max Activating Token Index: 425

Text #5

Max Range: 3.4005. Min Range: -3.4005

Max Act: 2.9328. Min Act: -0.0000

Data Index: 1501850 (Python Code)

Max Activating Token Index: 298

Text #6

Max Range: 3.4005. Min Range: -3.4005

Max Act: 2.8959. Min Act: -0.0000

Data Index: 1377773 (Python Code)

Max Activating Token Index: 722

Text #7

Max Range: 3.4005. Min Range: -3.4005

Max Act: 2.8930. Min Act: -0.0000

Data Index: 1439435 (Python Code)

Max Activating Token Index: 83

Text #8

Max Range: 3.4005. Min Range: -3.4005

Max Act: 2.8577. Min Act: -0.0000

Data Index: 1565354 (Python Code)

Max Activating Token Index: 493

Text #9

Max Range: 3.4005. Min Range: -3.4005

Max Act: 2.8436. Min Act: -0.0000

Data Index: 1530911 (Python Code)

Max Activating Token Index: 88

Text #10

Max Range: 3.4005. Min Range: -3.4005

Max Act: 2.8201. Min Act: -0.0000

Data Index: 1531405 (Python Code)

Max Activating Token Index: 104

Text #11

Max Range: 3.4005. Min Range: -3.4005

Max Act: 2.9427. Min Act: -0.0000

Data Index: 1369348 (Python Code)

Max Activating Token Index: 625

Text #12

Max Range: 3.4005. Min Range: -3.4005

Max Act: 2.8089. Min Act: -0.0000

Data Index: 1654182 (Python Code)

Max Activating Token Index: 977

Text #13

Max Range: 3.4005. Min Range: -3.4005

Max Act: 2.7744. Min Act: -0.0000

Data Index: 1513987 (Python Code)

Max Activating Token Index: 258

Text #14

Max Range: 3.4005. Min Range: -3.4005

Max Act: 2.6235. Min Act: -0.0000

Data Index: 1457879 (Python Code)

Max Activating Token Index: 758

Text #15

Max Range: 3.4005. Min Range: -3.4005

Max Act: 2.7511. Min Act: -0.0000

Data Index: 1378490 (Python Code)

Max Activating Token Index: 917

Text #16

Max Range: 3.4005. Min Range: -3.4005

Max Act: 2.5737. Min Act: -0.0000

Data Index: 1561459 (Python Code)

Max Activating Token Index: 582

Text #17

Max Range: 3.4005. Min Range: -3.4005

Max Act: 2.5712. Min Act: -0.0000

Data Index: 1598282 (Python Code)

Max Activating Token Index: 793

Text #18

Max Range: 3.4005. Min Range: -3.4005

Max Act: 2.6942. Min Act: -0.0000

Data Index: 1434944 (Python Code)

Max Activating Token Index: 706

Text #19

Max Range: 3.4005. Min Range: -3.4005

Max Act: 2.6861. Min Act: -0.0000

Data Index: 1529767 (Python Code)

Max Activating Token Index: 770