Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1683 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.5923. Min Range: -1.5923

Max Act: 1.5923. Min Act: -0.0000

Data Index: 1338855 (C4 (Web Text))

Max Activating Token Index: 541

Text #1

Max Range: 1.5923. Min Range: -1.5923

Max Act: 1.2495. Min Act: -0.0000

Data Index: 765915 (C4 (Web Text))

Max Activating Token Index: 714

Text #2

Max Range: 1.5923. Min Range: -1.5923

Max Act: 1.1534. Min Act: -0.0000

Data Index: 893011 (C4 (Web Text))

Max Activating Token Index: 547

Text #3

Max Range: 1.5923. Min Range: -1.5923

Max Act: 1.0923. Min Act: -0.0000

Data Index: 1217228 (C4 (Web Text))

Max Activating Token Index: 176

Text #4

Max Range: 1.5923. Min Range: -1.5923

Max Act: 1.1486. Min Act: -0.0000

Data Index: 507916 (C4 (Web Text))

Max Activating Token Index: 121

Text #5

Max Range: 1.5923. Min Range: -1.5923

Max Act: 1.1879. Min Act: -0.0000

Data Index: 643313 (C4 (Web Text))

Max Activating Token Index: 812

Text #6

Max Range: 1.5923. Min Range: -1.5923

Max Act: 1.1501. Min Act: -0.0000

Data Index: 1006047 (C4 (Web Text))

Max Activating Token Index: 499

Text #7

Max Range: 1.5923. Min Range: -1.5923

Max Act: 1.1425. Min Act: -0.0000

Data Index: 984919 (C4 (Web Text))

Max Activating Token Index: 800

Text #8

Max Range: 1.5923. Min Range: -1.5923

Max Act: 1.0477. Min Act: -0.0000

Data Index: 1131397 (C4 (Web Text))

Max Activating Token Index: 543

Text #9

Max Range: 1.5923. Min Range: -1.5923

Max Act: 1.0775. Min Act: -0.0000

Data Index: 564942 (C4 (Web Text))

Max Activating Token Index: 675

Text #10

Max Range: 1.5923. Min Range: -1.5923

Max Act: 1.1655. Min Act: -0.0000

Data Index: 727745 (C4 (Web Text))

Max Activating Token Index: 536

Text #11

Max Range: 1.5923. Min Range: -1.5923

Max Act: 1.1291. Min Act: -0.0000

Data Index: 959571 (C4 (Web Text))

Max Activating Token Index: 21

Text #12

Max Range: 1.5923. Min Range: -1.5923

Max Act: 1.0515. Min Act: -0.0000

Data Index: 971679 (C4 (Web Text))

Max Activating Token Index: 173

Text #13

Max Range: 1.5923. Min Range: -1.5923

Max Act: 0.9761. Min Act: -0.0000

Data Index: 1151187 (C4 (Web Text))

Max Activating Token Index: 636

Text #14

Max Range: 1.5923. Min Range: -1.5923

Max Act: 1.0659. Min Act: -0.0000

Data Index: 788515 (C4 (Web Text))

Max Activating Token Index: 810

Text #15

Max Range: 1.5923. Min Range: -1.5923

Max Act: 1.0302. Min Act: -0.0000

Data Index: 1097828 (C4 (Web Text))

Max Activating Token Index: 220

Text #16

Max Range: 1.5923. Min Range: -1.5923

Max Act: 1.0893. Min Act: -0.0000

Data Index: 1032748 (C4 (Web Text))

Max Activating Token Index: 590

Text #17

Max Range: 1.5923. Min Range: -1.5923

Max Act: 1.0383. Min Act: -0.0000

Data Index: 6456 (C4 (Web Text))

Max Activating Token Index: 695

Text #18

Max Range: 1.5923. Min Range: -1.5923

Max Act: 1.0059. Min Act: -0.0000

Data Index: 786382 (C4 (Web Text))

Max Activating Token Index: 772

Text #19

Max Range: 1.5923. Min Range: -1.5923

Max Act: 1.0469. Min Act: -0.0000

Data Index: 29053 (C4 (Web Text))

Max Activating Token Index: 996