Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3192 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.4731. Min Range: -0.4731

Max Act: 0.4703. Min Act: -0.0000

Data Index: 768697 (C4 (Web Text))

Max Activating Token Index: 271

Text #1

Max Range: 0.4731. Min Range: -0.4731

Max Act: 0.4377. Min Act: -0.0000

Data Index: 324280 (C4 (Web Text))

Max Activating Token Index: 608

Text #2

Max Range: 0.4731. Min Range: -0.4731

Max Act: 0.4731. Min Act: -0.0000

Data Index: 122102 (C4 (Web Text))

Max Activating Token Index: 103

Text #3

Max Range: 0.4731. Min Range: -0.4731

Max Act: 0.4609. Min Act: -0.0000

Data Index: 1050959 (C4 (Web Text))

Max Activating Token Index: 35

Text #4

Max Range: 0.4731. Min Range: -0.4731

Max Act: 0.4464. Min Act: -0.0000

Data Index: 505610 (C4 (Web Text))

Max Activating Token Index: 438

Text #5

Max Range: 0.4731. Min Range: -0.4731

Max Act: 0.3315. Min Act: -0.0000

Data Index: 431833 (C4 (Web Text))

Max Activating Token Index: 807

Text #6

Max Range: 0.4731. Min Range: -0.4731

Max Act: 0.3248. Min Act: -0.0000

Data Index: 270861 (C4 (Web Text))

Max Activating Token Index: 996

Text #7

Max Range: 0.4731. Min Range: -0.4731

Max Act: 0.3178. Min Act: -0.0000

Data Index: 934221 (C4 (Web Text))

Max Activating Token Index: 362

Text #8

Max Range: 0.4731. Min Range: -0.4731

Max Act: 0.3210. Min Act: -0.0000

Data Index: 37688 (C4 (Web Text))

Max Activating Token Index: 996

Text #9

Max Range: 0.4731. Min Range: -0.4731

Max Act: 0.3194. Min Act: -0.0000

Data Index: 984923 (C4 (Web Text))

Max Activating Token Index: 505

Text #10

Max Range: 0.4731. Min Range: -0.4731

Max Act: 0.3062. Min Act: -0.0000

Data Index: 1290490 (C4 (Web Text))

Max Activating Token Index: 662

Text #11

Max Range: 0.4731. Min Range: -0.4731

Max Act: 0.3051. Min Act: -0.0000

Data Index: 749534 (C4 (Web Text))

Max Activating Token Index: 132

Text #12

Max Range: 0.4731. Min Range: -0.4731

Max Act: 0.3092. Min Act: -0.0000

Data Index: 1249450 (C4 (Web Text))

Max Activating Token Index: 677

Text #13

Max Range: 0.4731. Min Range: -0.4731

Max Act: 0.2879. Min Act: -0.0000

Data Index: 1160802 (C4 (Web Text))

Max Activating Token Index: 510

Text #14

Max Range: 0.4731. Min Range: -0.4731

Max Act: 0.2806. Min Act: -0.0000

Data Index: 58101 (C4 (Web Text))

Max Activating Token Index: 232

Text #15

Max Range: 0.4731. Min Range: -0.4731

Max Act: 0.2745. Min Act: -0.0000

Data Index: 224384 (C4 (Web Text))

Max Activating Token Index: 219

Text #16

Max Range: 0.4731. Min Range: -0.4731

Max Act: 0.2607. Min Act: -0.0000

Data Index: 3905 (C4 (Web Text))

Max Activating Token Index: 946

Text #17

Max Range: 0.4731. Min Range: -0.4731

Max Act: 0.2801. Min Act: -0.0000

Data Index: 814437 (C4 (Web Text))

Max Activating Token Index: 158

Text #18

Max Range: 0.4731. Min Range: -0.4731

Max Act: 0.2898. Min Act: -0.0000

Data Index: 148805 (C4 (Web Text))

Max Activating Token Index: 707

Text #19

Max Range: 0.4731. Min Range: -0.4731

Max Act: 0.2884. Min Act: -0.0000

Data Index: 577636 (C4 (Web Text))

Max Activating Token Index: 609