Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 614 in Layer 1

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.1478. Min Range: -0.1478

Max Act: 0.1478. Min Act: -0.0002

Data Index: 285484 (C4 (Web Text))

Max Activating Token Index: 545

Text #1

Max Range: 0.1478. Min Range: -0.1478

Max Act: 0.1386. Min Act: -0.0002

Data Index: 1117786 (C4 (Web Text))

Max Activating Token Index: 967

Text #2

Max Range: 0.1478. Min Range: -0.1478

Max Act: 0.1353. Min Act: -0.0002

Data Index: 751083 (C4 (Web Text))

Max Activating Token Index: 321

Text #3

Max Range: 0.1478. Min Range: -0.1478

Max Act: 0.1416. Min Act: -0.0002

Data Index: 594125 (C4 (Web Text))

Max Activating Token Index: 319

Text #4

Max Range: 0.1478. Min Range: -0.1478

Max Act: 0.1339. Min Act: -0.0003

Data Index: 194836 (C4 (Web Text))

Max Activating Token Index: 605

Text #5

Max Range: 0.1478. Min Range: -0.1478

Max Act: 0.1336. Min Act: -0.0003

Data Index: 246998 (C4 (Web Text))

Max Activating Token Index: 119

Text #6

Max Range: 0.1478. Min Range: -0.1478

Max Act: 0.1329. Min Act: -0.0003

Data Index: 19465 (C4 (Web Text))

Max Activating Token Index: 723

Text #7

Max Range: 0.1478. Min Range: -0.1478

Max Act: 0.1328. Min Act: -0.0002

Data Index: 692719 (C4 (Web Text))

Max Activating Token Index: 477

Text #8

Max Range: 0.1478. Min Range: -0.1478

Max Act: 0.1304. Min Act: -0.0003

Data Index: 842953 (C4 (Web Text))

Max Activating Token Index: 331

Text #9

Max Range: 0.1478. Min Range: -0.1478

Max Act: 0.1286. Min Act: -0.0003

Data Index: 470863 (C4 (Web Text))

Max Activating Token Index: 710

Text #10

Max Range: 0.1478. Min Range: -0.1478

Max Act: 0.1306. Min Act: -0.0003

Data Index: 1228243 (C4 (Web Text))

Max Activating Token Index: 949

Text #11

Max Range: 0.1478. Min Range: -0.1478

Max Act: 0.1299. Min Act: -0.0002

Data Index: 828614 (C4 (Web Text))

Max Activating Token Index: 342

Text #12

Max Range: 0.1478. Min Range: -0.1478

Max Act: 0.1347. Min Act: -0.0002

Data Index: 342341 (C4 (Web Text))

Max Activating Token Index: 649

Text #13

Max Range: 0.1478. Min Range: -0.1478

Max Act: 0.1252. Min Act: -0.0003

Data Index: 19326 (C4 (Web Text))

Max Activating Token Index: 646

Text #14

Max Range: 0.1478. Min Range: -0.1478

Max Act: 0.1297. Min Act: -0.0003

Data Index: 469856 (C4 (Web Text))

Max Activating Token Index: 707

Text #15

Max Range: 0.1478. Min Range: -0.1478

Max Act: 0.1292. Min Act: -0.0002

Data Index: 553278 (C4 (Web Text))

Max Activating Token Index: 704

Text #16

Max Range: 0.1478. Min Range: -0.1478

Max Act: 0.1283. Min Act: -0.0002

Data Index: 1249526 (C4 (Web Text))

Max Activating Token Index: 499

Text #17

Max Range: 0.1478. Min Range: -0.1478

Max Act: 0.1272. Min Act: -0.0003

Data Index: 417657 (C4 (Web Text))

Max Activating Token Index: 612

Text #18

Max Range: 0.1478. Min Range: -0.1478

Max Act: 0.1179. Min Act: -0.0002

Data Index: 1239947 (C4 (Web Text))

Max Activating Token Index: 271

Text #19

Max Range: 0.1478. Min Range: -0.1478

Max Act: 0.1267. Min Act: -0.0003

Data Index: 1276857 (C4 (Web Text))

Max Activating Token Index: 724