Model: SoLU Model: 3 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1479 in Layer 1

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-3l')

Text #0

Max Range: 0.0466. Min Range: -0.0466

Max Act: 0.0466. Min Act: -0.0006

Data Index: 1626340 (Python Code)

Max Activating Token Index: 440

Text #1

Max Range: 0.0466. Min Range: -0.0466

Max Act: 0.0445. Min Act: -0.0006

Data Index: 1610998 (Python Code)

Max Activating Token Index: 364

Text #2

Max Range: 0.0466. Min Range: -0.0466

Max Act: 0.0427. Min Act: -0.0006

Data Index: 1409418 (Python Code)

Max Activating Token Index: 910

Text #3

Max Range: 0.0466. Min Range: -0.0466

Max Act: 0.0428. Min Act: -0.0006

Data Index: 1561660 (Python Code)

Max Activating Token Index: 329

Text #4

Max Range: 0.0466. Min Range: -0.0466

Max Act: 0.0437. Min Act: -0.0006

Data Index: 1443638 (Python Code)

Max Activating Token Index: 581

Text #5

Max Range: 0.0466. Min Range: -0.0466

Max Act: 0.0417. Min Act: -0.0006

Data Index: 1582677 (Python Code)

Max Activating Token Index: 789

Text #6

Max Range: 0.0466. Min Range: -0.0466

Max Act: 0.0434. Min Act: -0.0006

Data Index: 1386453 (Python Code)

Max Activating Token Index: 646

Text #7

Max Range: 0.0466. Min Range: -0.0466

Max Act: 0.0399. Min Act: -0.0006

Data Index: 1489974 (Python Code)

Max Activating Token Index: 623

Text #8

Max Range: 0.0466. Min Range: -0.0466

Max Act: 0.0427. Min Act: -0.0006

Data Index: 1501310 (Python Code)

Max Activating Token Index: 220

Text #9

Max Range: 0.0466. Min Range: -0.0466

Max Act: 0.0434. Min Act: -0.0006

Data Index: 1432303 (Python Code)

Max Activating Token Index: 449

Text #10

Max Range: 0.0466. Min Range: -0.0466

Max Act: 0.0403. Min Act: -0.0006

Data Index: 1566037 (Python Code)

Max Activating Token Index: 304

Text #11

Max Range: 0.0466. Min Range: -0.0466

Max Act: 0.0415. Min Act: -0.0006

Data Index: 1553620 (Python Code)

Max Activating Token Index: 585

Text #12

Max Range: 0.0466. Min Range: -0.0466

Max Act: 0.0403. Min Act: -0.0006

Data Index: 1545384 (Python Code)

Max Activating Token Index: 333

Text #13

Max Range: 0.0466. Min Range: -0.0466

Max Act: 0.0385. Min Act: -0.0006

Data Index: 1468654 (Python Code)

Max Activating Token Index: 341

Text #14

Max Range: 0.0466. Min Range: -0.0466

Max Act: 0.0387. Min Act: -0.0006

Data Index: 1419066 (Python Code)

Max Activating Token Index: 492

Text #15

Max Range: 0.0466. Min Range: -0.0466

Max Act: 0.0394. Min Act: -0.0006

Data Index: 1467447 (Python Code)

Max Activating Token Index: 799

Text #16

Max Range: 0.0466. Min Range: -0.0466

Max Act: 0.0393. Min Act: -0.0006

Data Index: 1433049 (Python Code)

Max Activating Token Index: 465

Text #17

Max Range: 0.0466. Min Range: -0.0466

Max Act: 0.0384. Min Act: -0.0006

Data Index: 1434232 (Python Code)

Max Activating Token Index: 484

Text #18

Max Range: 0.0466. Min Range: -0.0466

Max Act: 0.0397. Min Act: -0.0006

Data Index: 1561332 (Python Code)

Max Activating Token Index: 50

Text #19

Max Range: 0.0466. Min Range: -0.0466

Max Act: 0.0404. Min Act: -0.0006

Data Index: 1632843 (Python Code)

Max Activating Token Index: 433