Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4253 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.2670. Min Range: -1.2670

Max Act: 1.2670. Min Act: -0.0000

Data Index: 149988 (C4 (Web Text))

Max Activating Token Index: 440

Text #1

Max Range: 1.2670. Min Range: -1.2670

Max Act: 1.1245. Min Act: -0.0000

Data Index: 407660 (C4 (Web Text))

Max Activating Token Index: 828

Text #2

Max Range: 1.2670. Min Range: -1.2670

Max Act: 1.1915. Min Act: -0.0000

Data Index: 175980 (C4 (Web Text))

Max Activating Token Index: 1007

Text #3

Max Range: 1.2670. Min Range: -1.2670

Max Act: 1.1249. Min Act: -0.0000

Data Index: 683872 (C4 (Web Text))

Max Activating Token Index: 184

Text #4

Max Range: 1.2670. Min Range: -1.2670

Max Act: 1.1534. Min Act: -0.0000

Data Index: 463456 (C4 (Web Text))

Max Activating Token Index: 421

Text #5

Max Range: 1.2670. Min Range: -1.2670

Max Act: 1.0618. Min Act: -0.0000

Data Index: 174038 (C4 (Web Text))

Max Activating Token Index: 757

Text #6

Max Range: 1.2670. Min Range: -1.2670

Max Act: 1.0402. Min Act: -0.0000

Data Index: 78894 (C4 (Web Text))

Max Activating Token Index: 118

Text #7

Max Range: 1.2670. Min Range: -1.2670

Max Act: 1.0261. Min Act: -0.0000

Data Index: 650386 (C4 (Web Text))

Max Activating Token Index: 899

Text #8

Max Range: 1.2670. Min Range: -1.2670

Max Act: 0.8782. Min Act: -0.0000

Data Index: 624496 (C4 (Web Text))

Max Activating Token Index: 879

Text #9

Max Range: 1.2670. Min Range: -1.2670

Max Act: 0.9690. Min Act: -0.0000

Data Index: 127755 (C4 (Web Text))

Max Activating Token Index: 1012

Text #10

Max Range: 1.2670. Min Range: -1.2670

Max Act: 0.9468. Min Act: -0.0000

Data Index: 355733 (C4 (Web Text))

Max Activating Token Index: 930

Text #11

Max Range: 1.2670. Min Range: -1.2670

Max Act: 0.8998. Min Act: -0.0000

Data Index: 519461 (C4 (Web Text))

Max Activating Token Index: 352

Text #12

Max Range: 1.2670. Min Range: -1.2670

Max Act: 0.6084. Min Act: -0.0000

Data Index: 669461 (C4 (Web Text))

Max Activating Token Index: 1007

Text #13

Max Range: 1.2670. Min Range: -1.2670

Max Act: 0.5924. Min Act: -0.0000

Data Index: 48824 (C4 (Web Text))

Max Activating Token Index: 76

Text #14

Max Range: 1.2670. Min Range: -1.2670

Max Act: 0.4585. Min Act: -0.0000

Data Index: 794683 (C4 (Web Text))

Max Activating Token Index: 987

Text #15

Max Range: 1.2670. Min Range: -1.2670

Max Act: 0.4594. Min Act: -0.0000

Data Index: 378567 (C4 (Web Text))

Max Activating Token Index: 894

Text #16

Max Range: 1.2670. Min Range: -1.2670

Max Act: 0.4554. Min Act: -0.0000

Data Index: 430404 (C4 (Web Text))

Max Activating Token Index: 601

Text #17

Max Range: 1.2670. Min Range: -1.2670

Max Act: 0.4531. Min Act: -0.0000

Data Index: 693366 (C4 (Web Text))

Max Activating Token Index: 806

Text #18

Max Range: 1.2670. Min Range: -1.2670

Max Act: 0.4680. Min Act: -0.0000

Data Index: 958690 (C4 (Web Text))

Max Activating Token Index: 774

Text #19

Max Range: 1.2670. Min Range: -1.2670

Max Act: 0.4706. Min Act: -0.0000

Data Index: 1239526 (C4 (Web Text))

Max Activating Token Index: 927