Model: SoLU Model: 2 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 682 in Layer 1

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-2l')

Text #0

Max Range: 0.1601. Min Range: -0.1601

Max Act: 0.1601. Min Act: -0.0003

Data Index: 909822 (C4 (Web Text))

Max Activating Token Index: 610

Text #1

Max Range: 0.1601. Min Range: -0.1601

Max Act: 0.1135. Min Act: -0.0003

Data Index: 720607 (C4 (Web Text))

Max Activating Token Index: 177

Text #2

Max Range: 0.1601. Min Range: -0.1601

Max Act: 0.1071. Min Act: -0.0003

Data Index: 1091191 (C4 (Web Text))

Max Activating Token Index: 33

Text #3

Max Range: 0.1601. Min Range: -0.1601

Max Act: 0.1010. Min Act: -0.0003

Data Index: 1204931 (C4 (Web Text))

Max Activating Token Index: 906

Text #4

Max Range: 0.1601. Min Range: -0.1601

Max Act: 0.0995. Min Act: -0.0003

Data Index: 373881 (C4 (Web Text))

Max Activating Token Index: 121

Text #5

Max Range: 0.1601. Min Range: -0.1601

Max Act: 0.0958. Min Act: -0.0003

Data Index: 1297951 (C4 (Web Text))

Max Activating Token Index: 39

Text #6

Max Range: 0.1601. Min Range: -0.1601

Max Act: 0.0989. Min Act: -0.0003

Data Index: 1319432 (C4 (Web Text))

Max Activating Token Index: 594

Text #7

Max Range: 0.1601. Min Range: -0.1601

Max Act: 0.0971. Min Act: -0.0004

Data Index: 1050219 (C4 (Web Text))

Max Activating Token Index: 157

Text #8

Max Range: 0.1601. Min Range: -0.1601

Max Act: 0.0943. Min Act: -0.0003

Data Index: 795178 (C4 (Web Text))

Max Activating Token Index: 590

Text #9

Max Range: 0.1601. Min Range: -0.1601

Max Act: 0.0924. Min Act: -0.0003

Data Index: 816805 (C4 (Web Text))

Max Activating Token Index: 879

Text #10

Max Range: 0.1601. Min Range: -0.1601

Max Act: 0.0883. Min Act: -0.0003

Data Index: 110451 (C4 (Web Text))

Max Activating Token Index: 683

Text #11

Max Range: 0.1601. Min Range: -0.1601

Max Act: 0.0912. Min Act: -0.0003

Data Index: 824701 (C4 (Web Text))

Max Activating Token Index: 385

Text #12

Max Range: 0.1601. Min Range: -0.1601

Max Act: 0.0884. Min Act: -0.0003

Data Index: 1222642 (C4 (Web Text))

Max Activating Token Index: 635

Text #13

Max Range: 0.1601. Min Range: -0.1601

Max Act: 0.0899. Min Act: -0.0003

Data Index: 387858 (C4 (Web Text))

Max Activating Token Index: 871

Text #14

Max Range: 0.1601. Min Range: -0.1601

Max Act: 0.0880. Min Act: -0.0003

Data Index: 560564 (C4 (Web Text))

Max Activating Token Index: 86

Text #15

Max Range: 0.1601. Min Range: -0.1601

Max Act: 0.0917. Min Act: -0.0003

Data Index: 1072742 (C4 (Web Text))

Max Activating Token Index: 289

Text #16

Max Range: 0.1601. Min Range: -0.1601

Max Act: 0.0865. Min Act: -0.0003

Data Index: 798190 (C4 (Web Text))

Max Activating Token Index: 357

Text #17

Max Range: 0.1601. Min Range: -0.1601

Max Act: 0.0859. Min Act: -0.0002

Data Index: 1062203 (C4 (Web Text))

Max Activating Token Index: 208

Text #18

Max Range: 0.1601. Min Range: -0.1601

Max Act: 0.0844. Min Act: -0.0002

Data Index: 597312 (C4 (Web Text))

Max Activating Token Index: 672

Text #19

Max Range: 0.1601. Min Range: -0.1601

Max Act: 0.0845. Min Act: -0.0003

Data Index: 763127 (C4 (Web Text))

Max Activating Token Index: 132