Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3253 in Layer 4

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.1603. Min Range: -0.1603

Max Act: 0.1603. Min Act: -0.0001

Data Index: 1075815 (C4 (Web Text))

Max Activating Token Index: 112

Text #1

Max Range: 0.1603. Min Range: -0.1603

Max Act: 0.1473. Min Act: -0.0000

Data Index: 1633510 (Python Code)

Max Activating Token Index: 294

Text #2

Max Range: 0.1603. Min Range: -0.1603

Max Act: 0.1477. Min Act: -0.0000

Data Index: 1607069 (Python Code)

Max Activating Token Index: 666

Text #3

Max Range: 0.1603. Min Range: -0.1603

Max Act: 0.1350. Min Act: -0.0000

Data Index: 1540235 (Python Code)

Max Activating Token Index: 986

Text #4

Max Range: 0.1603. Min Range: -0.1603

Max Act: 0.1181. Min Act: -0.0001

Data Index: 436423 (C4 (Web Text))

Max Activating Token Index: 217

Text #5

Max Range: 0.1603. Min Range: -0.1603

Max Act: 0.1255. Min Act: -0.0000

Data Index: 1436128 (Python Code)

Max Activating Token Index: 605

Text #6

Max Range: 0.1603. Min Range: -0.1603

Max Act: 0.1200. Min Act: -0.0000

Data Index: 1573044 (Python Code)

Max Activating Token Index: 158

Text #7

Max Range: 0.1603. Min Range: -0.1603

Max Act: 0.1319. Min Act: -0.0001

Data Index: 1521521 (Python Code)

Max Activating Token Index: 18

Text #8

Max Range: 0.1603. Min Range: -0.1603

Max Act: 0.1128. Min Act: -0.0001

Data Index: 1582580 (Python Code)

Max Activating Token Index: 255

Text #9

Max Range: 0.1603. Min Range: -0.1603

Max Act: 0.1180. Min Act: -0.0001

Data Index: 1619700 (Python Code)

Max Activating Token Index: 86

Text #10

Max Range: 0.1603. Min Range: -0.1603

Max Act: 0.1199. Min Act: -0.0001

Data Index: 1274590 (C4 (Web Text))

Max Activating Token Index: 263

Text #11

Max Range: 0.1603. Min Range: -0.1603

Max Act: 0.1223. Min Act: -0.0000

Data Index: 1319255 (C4 (Web Text))

Max Activating Token Index: 75

Text #12

Max Range: 0.1603. Min Range: -0.1603

Max Act: 0.1273. Min Act: -0.0000

Data Index: 564401 (C4 (Web Text))

Max Activating Token Index: 318

Text #13

Max Range: 0.1603. Min Range: -0.1603

Max Act: 0.1041. Min Act: -0.0000

Data Index: 1424499 (Python Code)

Max Activating Token Index: 152

Text #14

Max Range: 0.1603. Min Range: -0.1603

Max Act: 0.1225. Min Act: -0.0001

Data Index: 1483737 (Python Code)

Max Activating Token Index: 125

Text #15

Max Range: 0.1603. Min Range: -0.1603

Max Act: 0.1219. Min Act: -0.0000

Data Index: 1630296 (Python Code)

Max Activating Token Index: 712

Text #16

Max Range: 0.1603. Min Range: -0.1603

Max Act: 0.1156. Min Act: -0.0001

Data Index: 1562360 (Python Code)

Max Activating Token Index: 11

Text #17

Max Range: 0.1603. Min Range: -0.1603

Max Act: 0.1182. Min Act: -0.0001

Data Index: 1505323 (Python Code)

Max Activating Token Index: 12

Text #18

Max Range: 0.1603. Min Range: -0.1603

Max Act: 0.1220. Min Act: -0.0000

Data Index: 1508622 (Python Code)

Max Activating Token Index: 41

Text #19

Max Range: 0.1603. Min Range: -0.1603

Max Act: 0.0993. Min Act: -0.0001

Data Index: 1618918 (Python Code)

Max Activating Token Index: 248