Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3764 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.1447. Min Range: -0.1447

Max Act: 0.1447. Min Act: -0.0002

Data Index: 1583195 (Python Code)

Max Activating Token Index: 104

Text #1

Max Range: 0.1447. Min Range: -0.1447

Max Act: 0.1253. Min Act: -0.0002

Data Index: 1648637 (Python Code)

Max Activating Token Index: 761

Text #2

Max Range: 0.1447. Min Range: -0.1447

Max Act: 0.1337. Min Act: -0.0002

Data Index: 1641535 (Python Code)

Max Activating Token Index: 141

Text #3

Max Range: 0.1447. Min Range: -0.1447

Max Act: 0.1241. Min Act: -0.0002

Data Index: 1392348 (Python Code)

Max Activating Token Index: 401

Text #4

Max Range: 0.1447. Min Range: -0.1447

Max Act: 0.1346. Min Act: -0.0002

Data Index: 1568987 (Python Code)

Max Activating Token Index: 392

Text #5

Max Range: 0.1447. Min Range: -0.1447

Max Act: 0.1262. Min Act: -0.0002

Data Index: 1572497 (Python Code)

Max Activating Token Index: 366

Text #6

Max Range: 0.1447. Min Range: -0.1447

Max Act: 0.1217. Min Act: -0.0002

Data Index: 1641607 (Python Code)

Max Activating Token Index: 889

Text #7

Max Range: 0.1447. Min Range: -0.1447

Max Act: 0.1303. Min Act: -0.0002

Data Index: 1509819 (Python Code)

Max Activating Token Index: 60

Text #8

Max Range: 0.1447. Min Range: -0.1447

Max Act: 0.1222. Min Act: -0.0002

Data Index: 1465340 (Python Code)

Max Activating Token Index: 987

Text #9

Max Range: 0.1447. Min Range: -0.1447

Max Act: 0.1152. Min Act: -0.0002

Data Index: 1532143 (Python Code)

Max Activating Token Index: 262

Text #10

Max Range: 0.1447. Min Range: -0.1447

Max Act: 0.1239. Min Act: -0.0002

Data Index: 1487043 (Python Code)

Max Activating Token Index: 116

Text #11

Max Range: 0.1447. Min Range: -0.1447

Max Act: 0.1206. Min Act: -0.0002

Data Index: 1583967 (Python Code)

Max Activating Token Index: 224

Text #12

Max Range: 0.1447. Min Range: -0.1447

Max Act: 0.1113. Min Act: -0.0002

Data Index: 920494 (C4 (Web Text))

Max Activating Token Index: 819

Text #13

Max Range: 0.1447. Min Range: -0.1447

Max Act: 0.1170. Min Act: -0.0002

Data Index: 1458741 (Python Code)

Max Activating Token Index: 849

Text #14

Max Range: 0.1447. Min Range: -0.1447

Max Act: 0.1189. Min Act: -0.0002

Data Index: 1448352 (Python Code)

Max Activating Token Index: 518

Text #15

Max Range: 0.1447. Min Range: -0.1447

Max Act: 0.1161. Min Act: -0.0002

Data Index: 357963 (C4 (Web Text))

Max Activating Token Index: 359

Text #16

Max Range: 0.1447. Min Range: -0.1447

Max Act: 0.1190. Min Act: -0.0002

Data Index: 1642652 (Python Code)

Max Activating Token Index: 540

Text #17

Max Range: 0.1447. Min Range: -0.1447

Max Act: 0.1143. Min Act: -0.0002

Data Index: 1402577 (Python Code)

Max Activating Token Index: 28

Text #18

Max Range: 0.1447. Min Range: -0.1447

Max Act: 0.1062. Min Act: -0.0002

Data Index: 1495038 (Python Code)

Max Activating Token Index: 355

Text #19

Max Range: 0.1447. Min Range: -0.1447

Max Act: 0.1183. Min Act: -0.0002

Data Index: 1456278 (Python Code)

Max Activating Token Index: 129