Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 662 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.3226. Min Range: -0.3226

Max Act: 0.2852. Min Act: -0.0000

Data Index: 912710 (C4 (Web Text))

Max Activating Token Index: 629

Text #1

Max Range: 0.3226. Min Range: -0.3226

Max Act: 0.3069. Min Act: -0.0000

Data Index: 139986 (C4 (Web Text))

Max Activating Token Index: 230

Text #2

Max Range: 0.3226. Min Range: -0.3226

Max Act: 0.3226. Min Act: -0.0000

Data Index: 681681 (C4 (Web Text))

Max Activating Token Index: 336

Text #3

Max Range: 0.3226. Min Range: -0.3226

Max Act: 0.2867. Min Act: -0.0000

Data Index: 1084682 (C4 (Web Text))

Max Activating Token Index: 720

Text #4

Max Range: 0.3226. Min Range: -0.3226

Max Act: 0.2881. Min Act: -0.0000

Data Index: 600464 (C4 (Web Text))

Max Activating Token Index: 922

Text #5

Max Range: 0.3226. Min Range: -0.3226

Max Act: 0.2768. Min Act: -0.0000

Data Index: 126961 (C4 (Web Text))

Max Activating Token Index: 372

Text #6

Max Range: 0.3226. Min Range: -0.3226

Max Act: 0.2926. Min Act: -0.0000

Data Index: 184798 (C4 (Web Text))

Max Activating Token Index: 951

Text #7

Max Range: 0.3226. Min Range: -0.3226

Max Act: 0.2663. Min Act: -0.0000

Data Index: 185149 (C4 (Web Text))

Max Activating Token Index: 612

Text #8

Max Range: 0.3226. Min Range: -0.3226

Max Act: 0.3041. Min Act: -0.0000

Data Index: 1121810 (C4 (Web Text))

Max Activating Token Index: 202

Text #9

Max Range: 0.3226. Min Range: -0.3226

Max Act: 0.2915. Min Act: -0.0000

Data Index: 1265935 (C4 (Web Text))

Max Activating Token Index: 939

Text #10

Max Range: 0.3226. Min Range: -0.3226

Max Act: 0.3144. Min Act: -0.0000

Data Index: 712354 (C4 (Web Text))

Max Activating Token Index: 644

Text #11

Max Range: 0.3226. Min Range: -0.3226

Max Act: 0.2788. Min Act: -0.0000

Data Index: 928210 (C4 (Web Text))

Max Activating Token Index: 559

Text #12

Max Range: 0.3226. Min Range: -0.3226

Max Act: 0.2623. Min Act: -0.0000

Data Index: 477042 (C4 (Web Text))

Max Activating Token Index: 180

Text #13

Max Range: 0.3226. Min Range: -0.3226

Max Act: 0.2459. Min Act: -0.0000

Data Index: 1095147 (C4 (Web Text))

Max Activating Token Index: 134

Text #14

Max Range: 0.3226. Min Range: -0.3226

Max Act: 0.2683. Min Act: -0.0000

Data Index: 272988 (C4 (Web Text))

Max Activating Token Index: 884

Text #15

Max Range: 0.3226. Min Range: -0.3226

Max Act: 0.2570. Min Act: -0.0000

Data Index: 501049 (C4 (Web Text))

Max Activating Token Index: 433

Text #16

Max Range: 0.3226. Min Range: -0.3226

Max Act: 0.2373. Min Act: -0.0000

Data Index: 678806 (C4 (Web Text))

Max Activating Token Index: 99

Text #17

Max Range: 0.3226. Min Range: -0.3226

Max Act: 0.2438. Min Act: -0.0000

Data Index: 675021 (C4 (Web Text))

Max Activating Token Index: 956

Text #18

Max Range: 0.3226. Min Range: -0.3226

Max Act: 0.2386. Min Act: -0.0000

Data Index: 249744 (C4 (Web Text))

Max Activating Token Index: 80

Text #19

Max Range: 0.3226. Min Range: -0.3226

Max Act: 0.2389. Min Act: -0.0000

Data Index: 633912 (C4 (Web Text))

Max Activating Token Index: 800