Model: SoLU Model: 2 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 881 in Layer 1

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-2l')

Text #0

Max Range: 0.1895. Min Range: -0.1895

Max Act: 0.1895. Min Act: -0.0003

Data Index: 596971 (C4 (Web Text))

Max Activating Token Index: 759

Text #1

Max Range: 0.1895. Min Range: -0.1895

Max Act: 0.1573. Min Act: -0.0003

Data Index: 235579 (C4 (Web Text))

Max Activating Token Index: 806

Text #2

Max Range: 0.1895. Min Range: -0.1895

Max Act: 0.1587. Min Act: -0.0003

Data Index: 812183 (C4 (Web Text))

Max Activating Token Index: 789

Text #3

Max Range: 0.1895. Min Range: -0.1895

Max Act: 0.1612. Min Act: -0.0003

Data Index: 417041 (C4 (Web Text))

Max Activating Token Index: 1019

Text #4

Max Range: 0.1895. Min Range: -0.1895

Max Act: 0.1510. Min Act: -0.0003

Data Index: 1012413 (C4 (Web Text))

Max Activating Token Index: 474

Text #5

Max Range: 0.1895. Min Range: -0.1895

Max Act: 0.1506. Min Act: -0.0004

Data Index: 1082646 (C4 (Web Text))

Max Activating Token Index: 143

Text #6

Max Range: 0.1895. Min Range: -0.1895

Max Act: 0.1514. Min Act: -0.0003

Data Index: 629763 (C4 (Web Text))

Max Activating Token Index: 574

Text #7

Max Range: 0.1895. Min Range: -0.1895

Max Act: 0.1566. Min Act: -0.0003

Data Index: 68103 (C4 (Web Text))

Max Activating Token Index: 981

Text #8

Max Range: 0.1895. Min Range: -0.1895

Max Act: 0.1436. Min Act: -0.0003

Data Index: 600064 (C4 (Web Text))

Max Activating Token Index: 766

Text #9

Max Range: 0.1895. Min Range: -0.1895

Max Act: 0.1528. Min Act: -0.0003

Data Index: 1059963 (C4 (Web Text))

Max Activating Token Index: 488

Text #10

Max Range: 0.1895. Min Range: -0.1895

Max Act: 0.1509. Min Act: -0.0003

Data Index: 1083263 (C4 (Web Text))

Max Activating Token Index: 849

Text #11

Max Range: 0.1895. Min Range: -0.1895

Max Act: 0.1464. Min Act: -0.0003

Data Index: 738237 (C4 (Web Text))

Max Activating Token Index: 491

Text #12

Max Range: 0.1895. Min Range: -0.1895

Max Act: 0.1493. Min Act: -0.0003

Data Index: 1133757 (C4 (Web Text))

Max Activating Token Index: 935

Text #13

Max Range: 0.1895. Min Range: -0.1895

Max Act: 0.1476. Min Act: -0.0003

Data Index: 955379 (C4 (Web Text))

Max Activating Token Index: 283

Text #14

Max Range: 0.1895. Min Range: -0.1895

Max Act: 0.1448. Min Act: -0.0003

Data Index: 28851 (C4 (Web Text))

Max Activating Token Index: 822

Text #15

Max Range: 0.1895. Min Range: -0.1895

Max Act: 0.1396. Min Act: -0.0003

Data Index: 431425 (C4 (Web Text))

Max Activating Token Index: 689

Text #16

Max Range: 0.1895. Min Range: -0.1895

Max Act: 0.1351. Min Act: -0.0003

Data Index: 394621 (C4 (Web Text))

Max Activating Token Index: 113

Text #17

Max Range: 0.1895. Min Range: -0.1895

Max Act: 0.1443. Min Act: -0.0003

Data Index: 18848 (C4 (Web Text))

Max Activating Token Index: 810

Text #18

Max Range: 0.1895. Min Range: -0.1895

Max Act: 0.1384. Min Act: -0.0003

Data Index: 1164613 (C4 (Web Text))

Max Activating Token Index: 825

Text #19

Max Range: 0.1895. Min Range: -0.1895

Max Act: 0.1301. Min Act: -0.0003

Data Index: 1238356 (C4 (Web Text))

Max Activating Token Index: 341