Model: SoLU Model: 6 Layers, 3072 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 86 in Layer 1

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-6l')

Text #0

Max Range: 0.0278. Min Range: -0.0278

Max Act: 0.0278. Min Act: -0.0004

Data Index: 1598830 (Python Code)

Max Activating Token Index: 60

Text #1

Max Range: 0.0278. Min Range: -0.0278

Max Act: 0.0248. Min Act: -0.0004

Data Index: 1656137 (Python Code)

Max Activating Token Index: 626

Text #2

Max Range: 0.0278. Min Range: -0.0278

Max Act: 0.0238. Min Act: -0.0004

Data Index: 1652379 (Python Code)

Max Activating Token Index: 77

Text #3

Max Range: 0.0278. Min Range: -0.0278

Max Act: 0.0242. Min Act: -0.0004

Data Index: 1409358 (Python Code)

Max Activating Token Index: 120

Text #4

Max Range: 0.0278. Min Range: -0.0278

Max Act: 0.0243. Min Act: -0.0004

Data Index: 1495735 (Python Code)

Max Activating Token Index: 64

Text #5

Max Range: 0.0278. Min Range: -0.0278

Max Act: 0.0247. Min Act: -0.0004

Data Index: 1539476 (Python Code)

Max Activating Token Index: 82

Text #6

Max Range: 0.0278. Min Range: -0.0278

Max Act: 0.0235. Min Act: -0.0004

Data Index: 1484015 (Python Code)

Max Activating Token Index: 72

Text #7

Max Range: 0.0278. Min Range: -0.0278

Max Act: 0.0225. Min Act: -0.0004

Data Index: 1488470 (Python Code)

Max Activating Token Index: 503

Text #8

Max Range: 0.0278. Min Range: -0.0278

Max Act: 0.0234. Min Act: -0.0004

Data Index: 1394451 (Python Code)

Max Activating Token Index: 141

Text #9

Max Range: 0.0278. Min Range: -0.0278

Max Act: 0.0228. Min Act: -0.0004

Data Index: 1547670 (Python Code)

Max Activating Token Index: 57

Text #10

Max Range: 0.0278. Min Range: -0.0278

Max Act: 0.0234. Min Act: -0.0004

Data Index: 1591167 (Python Code)

Max Activating Token Index: 83

Text #11

Max Range: 0.0278. Min Range: -0.0278

Max Act: 0.0225. Min Act: -0.0004

Data Index: 1612042 (Python Code)

Max Activating Token Index: 595

Text #12

Max Range: 0.0278. Min Range: -0.0278

Max Act: 0.0229. Min Act: -0.0004

Data Index: 1615600 (Python Code)

Max Activating Token Index: 680

Text #13

Max Range: 0.0278. Min Range: -0.0278

Max Act: 0.0228. Min Act: -0.0004

Data Index: 1618761 (Python Code)

Max Activating Token Index: 594

Text #14

Max Range: 0.0278. Min Range: -0.0278

Max Act: 0.0224. Min Act: -0.0004

Data Index: 1594885 (Python Code)

Max Activating Token Index: 858

Text #15

Max Range: 0.0278. Min Range: -0.0278

Max Act: 0.0232. Min Act: -0.0004

Data Index: 1547122 (Python Code)

Max Activating Token Index: 859

Text #16

Max Range: 0.0278. Min Range: -0.0278

Max Act: 0.0220. Min Act: -0.0004

Data Index: 1424015 (Python Code)

Max Activating Token Index: 39

Text #17

Max Range: 0.0278. Min Range: -0.0278

Max Act: 0.0222. Min Act: -0.0004

Data Index: 1632455 (Python Code)

Max Activating Token Index: 489

Text #18

Max Range: 0.0278. Min Range: -0.0278

Max Act: 0.0235. Min Act: -0.0004

Data Index: 1420003 (Python Code)

Max Activating Token Index: 330

Text #19

Max Range: 0.0278. Min Range: -0.0278

Max Act: 0.0226. Min Act: -0.0004

Data Index: 1521231 (Python Code)

Max Activating Token Index: 19