Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1849 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.1629. Min Range: -0.1629

Max Act: 0.1623. Min Act: -0.0002

Data Index: 1113490 (C4 (Web Text))

Max Activating Token Index: 253

Text #1

Max Range: 0.1629. Min Range: -0.1629

Max Act: 0.1629. Min Act: -0.0002

Data Index: 879199 (C4 (Web Text))

Max Activating Token Index: 412

Text #2

Max Range: 0.1629. Min Range: -0.1629

Max Act: 0.1559. Min Act: -0.0002

Data Index: 152970 (C4 (Web Text))

Max Activating Token Index: 751

Text #3

Max Range: 0.1629. Min Range: -0.1629

Max Act: 0.1527. Min Act: -0.0002

Data Index: 134359 (C4 (Web Text))

Max Activating Token Index: 100

Text #4

Max Range: 0.1629. Min Range: -0.1629

Max Act: 0.1432. Min Act: -0.0002

Data Index: 281215 (C4 (Web Text))

Max Activating Token Index: 940

Text #5

Max Range: 0.1629. Min Range: -0.1629

Max Act: 0.1626. Min Act: -0.0002

Data Index: 561454 (C4 (Web Text))

Max Activating Token Index: 850

Text #6

Max Range: 0.1629. Min Range: -0.1629

Max Act: 0.1460. Min Act: -0.0002

Data Index: 871249 (C4 (Web Text))

Max Activating Token Index: 949

Text #7

Max Range: 0.1629. Min Range: -0.1629

Max Act: 0.1307. Min Act: -0.0002

Data Index: 83643 (C4 (Web Text))

Max Activating Token Index: 1011

Text #8

Max Range: 0.1629. Min Range: -0.1629

Max Act: 0.1277. Min Act: -0.0002

Data Index: 706631 (C4 (Web Text))

Max Activating Token Index: 242

Text #9

Max Range: 0.1629. Min Range: -0.1629

Max Act: 0.1199. Min Act: -0.0002

Data Index: 859706 (C4 (Web Text))

Max Activating Token Index: 693

Text #10

Max Range: 0.1629. Min Range: -0.1629

Max Act: 0.1309. Min Act: -0.0002

Data Index: 387806 (C4 (Web Text))

Max Activating Token Index: 264

Text #11

Max Range: 0.1629. Min Range: -0.1629

Max Act: 0.1311. Min Act: -0.0002

Data Index: 475950 (C4 (Web Text))

Max Activating Token Index: 96

Text #12

Max Range: 0.1629. Min Range: -0.1629

Max Act: 0.1132. Min Act: -0.0002

Data Index: 481750 (C4 (Web Text))

Max Activating Token Index: 307

Text #13

Max Range: 0.1629. Min Range: -0.1629

Max Act: 0.1351. Min Act: -0.0002

Data Index: 1049927 (C4 (Web Text))

Max Activating Token Index: 431

Text #14

Max Range: 0.1629. Min Range: -0.1629

Max Act: 0.1282. Min Act: -0.0002

Data Index: 603593 (C4 (Web Text))

Max Activating Token Index: 464

Text #15

Max Range: 0.1629. Min Range: -0.1629

Max Act: 0.1369. Min Act: -0.0002

Data Index: 1347281 (C4 (Web Text))

Max Activating Token Index: 1018

Text #16

Max Range: 0.1629. Min Range: -0.1629

Max Act: 0.1239. Min Act: -0.0002

Data Index: 768280 (C4 (Web Text))

Max Activating Token Index: 635

Text #17

Max Range: 0.1629. Min Range: -0.1629

Max Act: 0.1276. Min Act: -0.0002

Data Index: 781169 (C4 (Web Text))

Max Activating Token Index: 53

Text #18

Max Range: 0.1629. Min Range: -0.1629

Max Act: 0.1177. Min Act: -0.0002

Data Index: 823491 (C4 (Web Text))

Max Activating Token Index: 9

Text #19

Max Range: 0.1629. Min Range: -0.1629

Max Act: 0.1314. Min Act: -0.0002

Data Index: 682901 (C4 (Web Text))

Max Activating Token Index: 51