Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1982 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 2.4794. Min Range: -2.4794

Max Act: 2.4794. Min Act: -0.0000

Data Index: 653053 (C4 (Web Text))

Max Activating Token Index: 228

Text #1

Max Range: 2.4794. Min Range: -2.4794

Max Act: 2.2745. Min Act: -0.0000

Data Index: 1143683 (C4 (Web Text))

Max Activating Token Index: 891

Text #2

Max Range: 2.4794. Min Range: -2.4794

Max Act: 2.2035. Min Act: -0.0000

Data Index: 1354727 (C4 (Web Text))

Max Activating Token Index: 113

Text #3

Max Range: 2.4794. Min Range: -2.4794

Max Act: 2.1884. Min Act: -0.0000

Data Index: 1069754 (C4 (Web Text))

Max Activating Token Index: 75

Text #4

Max Range: 2.4794. Min Range: -2.4794

Max Act: 2.0028. Min Act: -0.0000

Data Index: 467725 (C4 (Web Text))

Max Activating Token Index: 294

Text #5

Max Range: 2.4794. Min Range: -2.4794

Max Act: 2.1138. Min Act: -0.0000

Data Index: 902956 (C4 (Web Text))

Max Activating Token Index: 761

Text #6

Max Range: 2.4794. Min Range: -2.4794

Max Act: 1.9797. Min Act: -0.0000

Data Index: 455865 (C4 (Web Text))

Max Activating Token Index: 358

Text #7

Max Range: 2.4794. Min Range: -2.4794

Max Act: 2.0935. Min Act: -0.0000

Data Index: 234111 (C4 (Web Text))

Max Activating Token Index: 299

Text #8

Max Range: 2.4794. Min Range: -2.4794

Max Act: 2.0651. Min Act: -0.0000

Data Index: 31442 (C4 (Web Text))

Max Activating Token Index: 350

Text #9

Max Range: 2.4794. Min Range: -2.4794

Max Act: 1.7982. Min Act: -0.0000

Data Index: 87566 (C4 (Web Text))

Max Activating Token Index: 178

Text #10

Max Range: 2.4794. Min Range: -2.4794

Max Act: 1.8663. Min Act: -0.0000

Data Index: 331876 (C4 (Web Text))

Max Activating Token Index: 610

Text #11

Max Range: 2.4794. Min Range: -2.4794

Max Act: 1.7473. Min Act: -0.0000

Data Index: 207582 (C4 (Web Text))

Max Activating Token Index: 899

Text #12

Max Range: 2.4794. Min Range: -2.4794

Max Act: 1.7402. Min Act: -0.0000

Data Index: 1228574 (C4 (Web Text))

Max Activating Token Index: 226

Text #13

Max Range: 2.4794. Min Range: -2.4794

Max Act: 1.9443. Min Act: -0.0000

Data Index: 842521 (C4 (Web Text))

Max Activating Token Index: 918

Text #14

Max Range: 2.4794. Min Range: -2.4794

Max Act: 1.7427. Min Act: -0.0000

Data Index: 650290 (C4 (Web Text))

Max Activating Token Index: 760

Text #15

Max Range: 2.4794. Min Range: -2.4794

Max Act: 1.7556. Min Act: -0.0000

Data Index: 267059 (C4 (Web Text))

Max Activating Token Index: 560

Text #16

Max Range: 2.4794. Min Range: -2.4794

Max Act: 1.7483. Min Act: -0.0000

Data Index: 137301 (C4 (Web Text))

Max Activating Token Index: 176

Text #17

Max Range: 2.4794. Min Range: -2.4794

Max Act: 1.7269. Min Act: -0.0000

Data Index: 1067793 (C4 (Web Text))

Max Activating Token Index: 217

Text #18

Max Range: 2.4794. Min Range: -2.4794

Max Act: 1.8247. Min Act: -0.0000

Data Index: 96480 (C4 (Web Text))

Max Activating Token Index: 561

Text #19

Max Range: 2.4794. Min Range: -2.4794

Max Act: 1.7109. Min Act: -0.0000

Data Index: 658148 (C4 (Web Text))

Max Activating Token Index: 258