Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1106 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 5.3406. Min Range: -5.3406

Max Act: 5.3406. Min Act: -0.0000

Data Index: 1313931 (C4 (Web Text))

Max Activating Token Index: 276

Text #1

Max Range: 5.3406. Min Range: -5.3406

Max Act: 3.8492. Min Act: -0.0000

Data Index: 184552 (C4 (Web Text))

Max Activating Token Index: 605

Text #2

Max Range: 5.3406. Min Range: -5.3406

Max Act: 4.0088. Min Act: -0.0000

Data Index: 634131 (C4 (Web Text))

Max Activating Token Index: 111

Text #3

Max Range: 5.3406. Min Range: -5.3406

Max Act: 4.0056. Min Act: -0.0000

Data Index: 406627 (C4 (Web Text))

Max Activating Token Index: 595

Text #4

Max Range: 5.3406. Min Range: -5.3406

Max Act: 4.0105. Min Act: -0.0000

Data Index: 633364 (C4 (Web Text))

Max Activating Token Index: 157

Text #5

Max Range: 5.3406. Min Range: -5.3406

Max Act: 4.1758. Min Act: -0.0000

Data Index: 887759 (C4 (Web Text))

Max Activating Token Index: 341

Text #6

Max Range: 5.3406. Min Range: -5.3406

Max Act: 4.1645. Min Act: -0.0000

Data Index: 1333584 (C4 (Web Text))

Max Activating Token Index: 545

Text #7

Max Range: 5.3406. Min Range: -5.3406

Max Act: 3.9531. Min Act: -0.0000

Data Index: 625657 (C4 (Web Text))

Max Activating Token Index: 829

Text #8

Max Range: 5.3406. Min Range: -5.3406

Max Act: 3.5716. Min Act: -0.0000

Data Index: 559441 (C4 (Web Text))

Max Activating Token Index: 724

Text #9

Max Range: 5.3406. Min Range: -5.3406

Max Act: 4.0510. Min Act: -0.0000

Data Index: 585591 (C4 (Web Text))

Max Activating Token Index: 821

Text #10

Max Range: 5.3406. Min Range: -5.3406

Max Act: 4.0478. Min Act: -0.0000

Data Index: 814239 (C4 (Web Text))

Max Activating Token Index: 536

Text #11

Max Range: 5.3406. Min Range: -5.3406

Max Act: 3.8337. Min Act: -0.0000

Data Index: 445346 (C4 (Web Text))

Max Activating Token Index: 113

Text #12

Max Range: 5.3406. Min Range: -5.3406

Max Act: 3.8261. Min Act: -0.0000

Data Index: 756076 (C4 (Web Text))

Max Activating Token Index: 209

Text #13

Max Range: 5.3406. Min Range: -5.3406

Max Act: 3.8066. Min Act: -0.0000

Data Index: 1296544 (C4 (Web Text))

Max Activating Token Index: 663

Text #14

Max Range: 5.3406. Min Range: -5.3406

Max Act: 3.6527. Min Act: -0.0000

Data Index: 158057 (C4 (Web Text))

Max Activating Token Index: 739

Text #15

Max Range: 5.3406. Min Range: -5.3406

Max Act: 3.6379. Min Act: -0.0000

Data Index: 749236 (C4 (Web Text))

Max Activating Token Index: 669

Text #16

Max Range: 5.3406. Min Range: -5.3406

Max Act: 3.5889. Min Act: -0.0000

Data Index: 515406 (C4 (Web Text))

Max Activating Token Index: 738

Text #17

Max Range: 5.3406. Min Range: -5.3406

Max Act: 3.7261. Min Act: -0.0000

Data Index: 576955 (C4 (Web Text))

Max Activating Token Index: 247

Text #18

Max Range: 5.3406. Min Range: -5.3406

Max Act: 3.4379. Min Act: -0.0000

Data Index: 1301164 (C4 (Web Text))

Max Activating Token Index: 946

Text #19

Max Range: 5.3406. Min Range: -5.3406

Max Act: 3.7252. Min Act: -0.0000

Data Index: 1134217 (C4 (Web Text))

Max Activating Token Index: 678