Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 519 in Layer 1

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.2028. Min Range: -0.2028

Max Act: 0.2028. Min Act: -0.0002

Data Index: 70453 (C4 (Web Text))

Max Activating Token Index: 249

Text #1

Max Range: 0.2028. Min Range: -0.2028

Max Act: 0.1811. Min Act: -0.0003

Data Index: 1436 (C4 (Web Text))

Max Activating Token Index: 863

Text #2

Max Range: 0.2028. Min Range: -0.2028

Max Act: 0.1596. Min Act: -0.0003

Data Index: 438883 (C4 (Web Text))

Max Activating Token Index: 339

Text #3

Max Range: 0.2028. Min Range: -0.2028

Max Act: 0.1453. Min Act: -0.0002

Data Index: 610875 (C4 (Web Text))

Max Activating Token Index: 301

Text #4

Max Range: 0.2028. Min Range: -0.2028

Max Act: 0.1492. Min Act: -0.0002

Data Index: 649704 (C4 (Web Text))

Max Activating Token Index: 66

Text #5

Max Range: 0.2028. Min Range: -0.2028

Max Act: 0.1472. Min Act: -0.0003

Data Index: 483346 (C4 (Web Text))

Max Activating Token Index: 313

Text #6

Max Range: 0.2028. Min Range: -0.2028

Max Act: 0.1389. Min Act: -0.0002

Data Index: 216786 (C4 (Web Text))

Max Activating Token Index: 336

Text #7

Max Range: 0.2028. Min Range: -0.2028

Max Act: 0.1392. Min Act: -0.0003

Data Index: 396619 (C4 (Web Text))

Max Activating Token Index: 829

Text #8

Max Range: 0.2028. Min Range: -0.2028

Max Act: 0.1387. Min Act: -0.0003

Data Index: 1250332 (C4 (Web Text))

Max Activating Token Index: 975

Text #9

Max Range: 0.2028. Min Range: -0.2028

Max Act: 0.1386. Min Act: -0.0002

Data Index: 265623 (C4 (Web Text))

Max Activating Token Index: 998

Text #10

Max Range: 0.2028. Min Range: -0.2028

Max Act: 0.1370. Min Act: -0.0002

Data Index: 605227 (C4 (Web Text))

Max Activating Token Index: 914

Text #11

Max Range: 0.2028. Min Range: -0.2028

Max Act: 0.1343. Min Act: -0.0003

Data Index: 797767 (C4 (Web Text))

Max Activating Token Index: 944

Text #12

Max Range: 0.2028. Min Range: -0.2028

Max Act: 0.1373. Min Act: -0.0003

Data Index: 1281095 (C4 (Web Text))

Max Activating Token Index: 180

Text #13

Max Range: 0.2028. Min Range: -0.2028

Max Act: 0.1335. Min Act: -0.0002

Data Index: 1192421 (C4 (Web Text))

Max Activating Token Index: 956

Text #14

Max Range: 0.2028. Min Range: -0.2028

Max Act: 0.1333. Min Act: -0.0002

Data Index: 99417 (C4 (Web Text))

Max Activating Token Index: 119

Text #15

Max Range: 0.2028. Min Range: -0.2028

Max Act: 0.1368. Min Act: -0.0003

Data Index: 324500 (C4 (Web Text))

Max Activating Token Index: 206

Text #16

Max Range: 0.2028. Min Range: -0.2028

Max Act: 0.1306. Min Act: -0.0003

Data Index: 922699 (C4 (Web Text))

Max Activating Token Index: 997

Text #17

Max Range: 0.2028. Min Range: -0.2028

Max Act: 0.1344. Min Act: -0.0003

Data Index: 1202304 (C4 (Web Text))

Max Activating Token Index: 121

Text #18

Max Range: 0.2028. Min Range: -0.2028

Max Act: 0.1251. Min Act: -0.0002

Data Index: 509265 (C4 (Web Text))

Max Activating Token Index: 225

Text #19

Max Range: 0.2028. Min Range: -0.2028

Max Act: 0.1289. Min Act: -0.0002

Data Index: 1103041 (C4 (Web Text))

Max Activating Token Index: 398