Model: SoLU Model: 6 Layers, 3072 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 995 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-6l')

Text #0

Max Range: 0.1779. Min Range: -0.1779

Max Act: 0.1779. Min Act: -0.0003

Data Index: 932117 (C4 (Web Text))

Max Activating Token Index: 831

Text #1

Max Range: 0.1779. Min Range: -0.1779

Max Act: 0.1423. Min Act: -0.0003

Data Index: 502231 (C4 (Web Text))

Max Activating Token Index: 910

Text #2

Max Range: 0.1779. Min Range: -0.1779

Max Act: 0.1631. Min Act: -0.0003

Data Index: 577135 (C4 (Web Text))

Max Activating Token Index: 152

Text #3

Max Range: 0.1779. Min Range: -0.1779

Max Act: 0.1505. Min Act: -0.0003

Data Index: 386447 (C4 (Web Text))

Max Activating Token Index: 401

Text #4

Max Range: 0.1779. Min Range: -0.1779

Max Act: 0.1495. Min Act: -0.0003

Data Index: 906960 (C4 (Web Text))

Max Activating Token Index: 895

Text #5

Max Range: 0.1779. Min Range: -0.1779

Max Act: 0.1580. Min Act: -0.0003

Data Index: 568042 (C4 (Web Text))

Max Activating Token Index: 255

Text #6

Max Range: 0.1779. Min Range: -0.1779

Max Act: 0.1468. Min Act: -0.0003

Data Index: 230033 (C4 (Web Text))

Max Activating Token Index: 690

Text #7

Max Range: 0.1779. Min Range: -0.1779

Max Act: 0.1450. Min Act: -0.0003

Data Index: 485010 (C4 (Web Text))

Max Activating Token Index: 988

Text #8

Max Range: 0.1779. Min Range: -0.1779

Max Act: 0.1389. Min Act: -0.0003

Data Index: 770184 (C4 (Web Text))

Max Activating Token Index: 302

Text #9

Max Range: 0.1779. Min Range: -0.1779

Max Act: 0.1464. Min Act: -0.0003

Data Index: 878314 (C4 (Web Text))

Max Activating Token Index: 923

Text #10

Max Range: 0.1779. Min Range: -0.1779

Max Act: 0.1499. Min Act: -0.0003

Data Index: 22817 (C4 (Web Text))

Max Activating Token Index: 146

Text #11

Max Range: 0.1779. Min Range: -0.1779

Max Act: 0.1441. Min Act: -0.0003

Data Index: 311772 (C4 (Web Text))

Max Activating Token Index: 57

Text #12

Max Range: 0.1779. Min Range: -0.1779

Max Act: 0.1320. Min Act: -0.0003

Data Index: 893998 (C4 (Web Text))

Max Activating Token Index: 296

Text #13

Max Range: 0.1779. Min Range: -0.1779

Max Act: 0.1484. Min Act: -0.0003

Data Index: 1210554 (C4 (Web Text))

Max Activating Token Index: 247

Text #14

Max Range: 0.1779. Min Range: -0.1779

Max Act: 0.1372. Min Act: -0.0003

Data Index: 248098 (C4 (Web Text))

Max Activating Token Index: 103

Text #15

Max Range: 0.1779. Min Range: -0.1779

Max Act: 0.1374. Min Act: -0.0003

Data Index: 572445 (C4 (Web Text))

Max Activating Token Index: 265

Text #16

Max Range: 0.1779. Min Range: -0.1779

Max Act: 0.1421. Min Act: -0.0003

Data Index: 1151991 (C4 (Web Text))

Max Activating Token Index: 182

Text #17

Max Range: 0.1779. Min Range: -0.1779

Max Act: 0.1348. Min Act: -0.0003

Data Index: 562229 (C4 (Web Text))

Max Activating Token Index: 666

Text #18

Max Range: 0.1779. Min Range: -0.1779

Max Act: 0.1472. Min Act: -0.0003

Data Index: 670315 (C4 (Web Text))

Max Activating Token Index: 215

Text #19

Max Range: 0.1779. Min Range: -0.1779

Max Act: 0.1508. Min Act: -0.0003

Data Index: 909733 (C4 (Web Text))

Max Activating Token Index: 367