Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 559 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.1895. Min Range: -0.1895

Max Act: 0.1895. Min Act: -0.0002

Data Index: 178719 (C4 (Web Text))

Max Activating Token Index: 879

Text #1

Max Range: 0.1895. Min Range: -0.1895

Max Act: 0.1788. Min Act: -0.0002

Data Index: 16637 (C4 (Web Text))

Max Activating Token Index: 97

Text #2

Max Range: 0.1895. Min Range: -0.1895

Max Act: 0.1675. Min Act: -0.0002

Data Index: 67302 (C4 (Web Text))

Max Activating Token Index: 209

Text #3

Max Range: 0.1895. Min Range: -0.1895

Max Act: 0.1698. Min Act: -0.0002

Data Index: 1082406 (C4 (Web Text))

Max Activating Token Index: 32

Text #4

Max Range: 0.1895. Min Range: -0.1895

Max Act: 0.1631. Min Act: -0.0002

Data Index: 128902 (C4 (Web Text))

Max Activating Token Index: 945

Text #5

Max Range: 0.1895. Min Range: -0.1895

Max Act: 0.1595. Min Act: -0.0002

Data Index: 1020443 (C4 (Web Text))

Max Activating Token Index: 42

Text #6

Max Range: 0.1895. Min Range: -0.1895

Max Act: 0.1599. Min Act: -0.0002

Data Index: 347373 (C4 (Web Text))

Max Activating Token Index: 174

Text #7

Max Range: 0.1895. Min Range: -0.1895

Max Act: 0.1664. Min Act: -0.0002

Data Index: 327436 (C4 (Web Text))

Max Activating Token Index: 2

Text #8

Max Range: 0.1895. Min Range: -0.1895

Max Act: 0.1457. Min Act: -0.0002

Data Index: 1345184 (C4 (Web Text))

Max Activating Token Index: 60

Text #9

Max Range: 0.1895. Min Range: -0.1895

Max Act: 0.1440. Min Act: -0.0002

Data Index: 363287 (C4 (Web Text))

Max Activating Token Index: 255

Text #10

Max Range: 0.1895. Min Range: -0.1895

Max Act: 0.1675. Min Act: -0.0002

Data Index: 1288454 (C4 (Web Text))

Max Activating Token Index: 195

Text #11

Max Range: 0.1895. Min Range: -0.1895

Max Act: 0.1435. Min Act: -0.0002

Data Index: 1296632 (C4 (Web Text))

Max Activating Token Index: 195

Text #12

Max Range: 0.1895. Min Range: -0.1895

Max Act: 0.1460. Min Act: -0.0002

Data Index: 1310599 (C4 (Web Text))

Max Activating Token Index: 463

Text #13

Max Range: 0.1895. Min Range: -0.1895

Max Act: 0.1551. Min Act: -0.0002

Data Index: 498949 (C4 (Web Text))

Max Activating Token Index: 15

Text #14

Max Range: 0.1895. Min Range: -0.1895

Max Act: 0.1382. Min Act: -0.0002

Data Index: 1226934 (C4 (Web Text))

Max Activating Token Index: 951

Text #15

Max Range: 0.1895. Min Range: -0.1895

Max Act: 0.1383. Min Act: -0.0002

Data Index: 945610 (C4 (Web Text))

Max Activating Token Index: 79

Text #16

Max Range: 0.1895. Min Range: -0.1895

Max Act: 0.1458. Min Act: -0.0002

Data Index: 380176 (C4 (Web Text))

Max Activating Token Index: 751

Text #17

Max Range: 0.1895. Min Range: -0.1895

Max Act: 0.1346. Min Act: -0.0002

Data Index: 83338 (C4 (Web Text))

Max Activating Token Index: 151

Text #18

Max Range: 0.1895. Min Range: -0.1895

Max Act: 0.1499. Min Act: -0.0002

Data Index: 1275106 (C4 (Web Text))

Max Activating Token Index: 536

Text #19

Max Range: 0.1895. Min Range: -0.1895

Max Act: 0.1499. Min Act: -0.0002

Data Index: 473354 (C4 (Web Text))

Max Activating Token Index: 968