Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1597 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.0986. Min Range: -0.0986

Max Act: 0.0986. Min Act: -0.0002

Data Index: 275862 (C4 (Web Text))

Max Activating Token Index: 516

Text #1

Max Range: 0.0986. Min Range: -0.0986

Max Act: 0.0890. Min Act: -0.0002

Data Index: 445980 (C4 (Web Text))

Max Activating Token Index: 362

Text #2

Max Range: 0.0986. Min Range: -0.0986

Max Act: 0.0852. Min Act: -0.0002

Data Index: 951679 (C4 (Web Text))

Max Activating Token Index: 233

Text #3

Max Range: 0.0986. Min Range: -0.0986

Max Act: 0.0809. Min Act: -0.0002

Data Index: 314184 (C4 (Web Text))

Max Activating Token Index: 147

Text #4

Max Range: 0.0986. Min Range: -0.0986

Max Act: 0.0807. Min Act: -0.0002

Data Index: 963835 (C4 (Web Text))

Max Activating Token Index: 423

Text #5

Max Range: 0.0986. Min Range: -0.0986

Max Act: 0.0794. Min Act: -0.0002

Data Index: 237497 (C4 (Web Text))

Max Activating Token Index: 650

Text #6

Max Range: 0.0986. Min Range: -0.0986

Max Act: 0.0813. Min Act: -0.0002

Data Index: 587031 (C4 (Web Text))

Max Activating Token Index: 920

Text #7

Max Range: 0.0986. Min Range: -0.0986

Max Act: 0.0737. Min Act: -0.0002

Data Index: 29144 (C4 (Web Text))

Max Activating Token Index: 734

Text #8

Max Range: 0.0986. Min Range: -0.0986

Max Act: 0.0724. Min Act: -0.0002

Data Index: 431766 (C4 (Web Text))

Max Activating Token Index: 442

Text #9

Max Range: 0.0986. Min Range: -0.0986

Max Act: 0.0684. Min Act: -0.0002

Data Index: 245626 (C4 (Web Text))

Max Activating Token Index: 681

Text #10

Max Range: 0.0986. Min Range: -0.0986

Max Act: 0.0756. Min Act: -0.0002

Data Index: 1293816 (C4 (Web Text))

Max Activating Token Index: 889

Text #11

Max Range: 0.0986. Min Range: -0.0986

Max Act: 0.0678. Min Act: -0.0002

Data Index: 738763 (C4 (Web Text))

Max Activating Token Index: 63

Text #12

Max Range: 0.0986. Min Range: -0.0986

Max Act: 0.0739. Min Act: -0.0002

Data Index: 529999 (C4 (Web Text))

Max Activating Token Index: 646

Text #13

Max Range: 0.0986. Min Range: -0.0986

Max Act: 0.0640. Min Act: -0.0002

Data Index: 1242288 (C4 (Web Text))

Max Activating Token Index: 359

Text #14

Max Range: 0.0986. Min Range: -0.0986

Max Act: 0.0652. Min Act: -0.0002

Data Index: 869054 (C4 (Web Text))

Max Activating Token Index: 934

Text #15

Max Range: 0.0986. Min Range: -0.0986

Max Act: 0.0612. Min Act: -0.0002

Data Index: 881526 (C4 (Web Text))

Max Activating Token Index: 315

Text #16

Max Range: 0.0986. Min Range: -0.0986

Max Act: 0.0583. Min Act: -0.0002

Data Index: 169515 (C4 (Web Text))

Max Activating Token Index: 71

Text #17

Max Range: 0.0986. Min Range: -0.0986

Max Act: 0.0600. Min Act: -0.0002

Data Index: 537545 (C4 (Web Text))

Max Activating Token Index: 138

Text #18

Max Range: 0.0986. Min Range: -0.0986

Max Act: 0.0640. Min Act: -0.0002

Data Index: 562968 (C4 (Web Text))

Max Activating Token Index: 66

Text #19

Max Range: 0.0986. Min Range: -0.0986

Max Act: 0.0611. Min Act: -0.0002

Data Index: 263356 (C4 (Web Text))

Max Activating Token Index: 670