Model: SoLU Model: 2 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1946 in Layer 1

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-2l')

Text #0

Max Range: 0.1534. Min Range: -0.1534

Max Act: 0.1534. Min Act: -0.0003

Data Index: 601322 (C4 (Web Text))

Max Activating Token Index: 90

Text #1

Max Range: 0.1534. Min Range: -0.1534

Max Act: 0.1508. Min Act: -0.0003

Data Index: 510071 (C4 (Web Text))

Max Activating Token Index: 650

Text #2

Max Range: 0.1534. Min Range: -0.1534

Max Act: 0.1444. Min Act: -0.0003

Data Index: 97970 (C4 (Web Text))

Max Activating Token Index: 194

Text #3

Max Range: 0.1534. Min Range: -0.1534

Max Act: 0.1449. Min Act: -0.0003

Data Index: 436497 (C4 (Web Text))

Max Activating Token Index: 496

Text #4

Max Range: 0.1534. Min Range: -0.1534

Max Act: 0.1413. Min Act: -0.0003

Data Index: 186254 (C4 (Web Text))

Max Activating Token Index: 271

Text #5

Max Range: 0.1534. Min Range: -0.1534

Max Act: 0.1374. Min Act: -0.0003

Data Index: 1188843 (C4 (Web Text))

Max Activating Token Index: 1007

Text #6

Max Range: 0.1534. Min Range: -0.1534

Max Act: 0.1365. Min Act: -0.0003

Data Index: 630043 (C4 (Web Text))

Max Activating Token Index: 1009

Text #7

Max Range: 0.1534. Min Range: -0.1534

Max Act: 0.1262. Min Act: -0.0003

Data Index: 440744 (C4 (Web Text))

Max Activating Token Index: 900

Text #8

Max Range: 0.1534. Min Range: -0.1534

Max Act: 0.1400. Min Act: -0.0003

Data Index: 138916 (C4 (Web Text))

Max Activating Token Index: 154

Text #9

Max Range: 0.1534. Min Range: -0.1534

Max Act: 0.1351. Min Act: -0.0003

Data Index: 1349774 (C4 (Web Text))

Max Activating Token Index: 191

Text #10

Max Range: 0.1534. Min Range: -0.1534

Max Act: 0.1353. Min Act: -0.0003

Data Index: 1240268 (C4 (Web Text))

Max Activating Token Index: 873

Text #11

Max Range: 0.1534. Min Range: -0.1534

Max Act: 0.1395. Min Act: -0.0003

Data Index: 414670 (C4 (Web Text))

Max Activating Token Index: 411

Text #12

Max Range: 0.1534. Min Range: -0.1534

Max Act: 0.1374. Min Act: -0.0003

Data Index: 735442 (C4 (Web Text))

Max Activating Token Index: 961

Text #13

Max Range: 0.1534. Min Range: -0.1534

Max Act: 0.1376. Min Act: -0.0003

Data Index: 1009507 (C4 (Web Text))

Max Activating Token Index: 87

Text #14

Max Range: 0.1534. Min Range: -0.1534

Max Act: 0.1231. Min Act: -0.0003

Data Index: 584447 (C4 (Web Text))

Max Activating Token Index: 300

Text #15

Max Range: 0.1534. Min Range: -0.1534

Max Act: 0.1401. Min Act: -0.0003

Data Index: 541186 (C4 (Web Text))

Max Activating Token Index: 874

Text #16

Max Range: 0.1534. Min Range: -0.1534

Max Act: 0.1306. Min Act: -0.0003

Data Index: 893998 (C4 (Web Text))

Max Activating Token Index: 200

Text #17

Max Range: 0.1534. Min Range: -0.1534

Max Act: 0.1294. Min Act: -0.0003

Data Index: 1232060 (C4 (Web Text))

Max Activating Token Index: 554

Text #18

Max Range: 0.1534. Min Range: -0.1534

Max Act: 0.1305. Min Act: -0.0003

Data Index: 180085 (C4 (Web Text))

Max Activating Token Index: 865

Text #19

Max Range: 0.1534. Min Range: -0.1534

Max Act: 0.1348. Min Act: -0.0003

Data Index: 216959 (C4 (Web Text))

Max Activating Token Index: 279