Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1966 in Layer 4

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.1549. Min Range: -0.1549

Max Act: 0.1528. Min Act: -0.0001

Data Index: 1634587 (Python Code)

Max Activating Token Index: 220

Text #1

Max Range: 0.1549. Min Range: -0.1549

Max Act: 0.1549. Min Act: -0.0001

Data Index: 1654821 (Python Code)

Max Activating Token Index: 403

Text #2

Max Range: 0.1549. Min Range: -0.1549

Max Act: 0.1356. Min Act: -0.0001

Data Index: 1599112 (Python Code)

Max Activating Token Index: 650

Text #3

Max Range: 0.1549. Min Range: -0.1549

Max Act: 0.1057. Min Act: -0.0001

Data Index: 1447892 (Python Code)

Max Activating Token Index: 108

Text #4

Max Range: 0.1549. Min Range: -0.1549

Max Act: 0.1187. Min Act: -0.0001

Data Index: 1637139 (Python Code)

Max Activating Token Index: 312

Text #5

Max Range: 0.1549. Min Range: -0.1549

Max Act: 0.1219. Min Act: -0.0001

Data Index: 1646009 (Python Code)

Max Activating Token Index: 741

Text #6

Max Range: 0.1549. Min Range: -0.1549

Max Act: 0.1170. Min Act: -0.0001

Data Index: 1515848 (Python Code)

Max Activating Token Index: 978

Text #7

Max Range: 0.1549. Min Range: -0.1549

Max Act: 0.1156. Min Act: -0.0001

Data Index: 1431131 (Python Code)

Max Activating Token Index: 966

Text #8

Max Range: 0.1549. Min Range: -0.1549

Max Act: 0.1144. Min Act: -0.0001

Data Index: 1422825 (Python Code)

Max Activating Token Index: 928

Text #9

Max Range: 0.1549. Min Range: -0.1549

Max Act: 0.1112. Min Act: -0.0001

Data Index: 1580274 (Python Code)

Max Activating Token Index: 287

Text #10

Max Range: 0.1549. Min Range: -0.1549

Max Act: 0.1172. Min Act: -0.0001

Data Index: 1453174 (Python Code)

Max Activating Token Index: 636

Text #11

Max Range: 0.1549. Min Range: -0.1549

Max Act: 0.0997. Min Act: -0.0001

Data Index: 1570761 (Python Code)

Max Activating Token Index: 936

Text #12

Max Range: 0.1549. Min Range: -0.1549

Max Act: 0.1188. Min Act: -0.0000

Data Index: 1539354 (Python Code)

Max Activating Token Index: 511

Text #13

Max Range: 0.1549. Min Range: -0.1549

Max Act: 0.1137. Min Act: -0.0001

Data Index: 241278 (C4 (Web Text))

Max Activating Token Index: 284

Text #14

Max Range: 0.1549. Min Range: -0.1549

Max Act: 0.1052. Min Act: -0.0001

Data Index: 1441486 (Python Code)

Max Activating Token Index: 615

Text #15

Max Range: 0.1549. Min Range: -0.1549

Max Act: 0.1055. Min Act: -0.0001

Data Index: 1607699 (Python Code)

Max Activating Token Index: 708

Text #16

Max Range: 0.1549. Min Range: -0.1549

Max Act: 0.1084. Min Act: -0.0001

Data Index: 1576140 (Python Code)

Max Activating Token Index: 567

Text #17

Max Range: 0.1549. Min Range: -0.1549

Max Act: 0.1087. Min Act: -0.0001

Data Index: 1640908 (Python Code)

Max Activating Token Index: 372

Text #18

Max Range: 0.1549. Min Range: -0.1549

Max Act: 0.1120. Min Act: -0.0000

Data Index: 1498643 (Python Code)

Max Activating Token Index: 571

Text #19

Max Range: 0.1549. Min Range: -0.1549

Max Act: 0.1002. Min Act: -0.0001

Data Index: 1429015 (Python Code)

Max Activating Token Index: 934