Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 820 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 3.5573. Min Range: -3.5573

Max Act: 3.5573. Min Act: -0.0000

Data Index: 732045 (C4 (Web Text))

Max Activating Token Index: 946

Text #1

Max Range: 3.5573. Min Range: -3.5573

Max Act: 2.8569. Min Act: -0.0000

Data Index: 225929 (C4 (Web Text))

Max Activating Token Index: 461

Text #2

Max Range: 3.5573. Min Range: -3.5573

Max Act: 2.6331. Min Act: -0.0000

Data Index: 210576 (C4 (Web Text))

Max Activating Token Index: 774

Text #3

Max Range: 3.5573. Min Range: -3.5573

Max Act: 2.7657. Min Act: -0.0000

Data Index: 161015 (C4 (Web Text))

Max Activating Token Index: 390

Text #4

Max Range: 3.5573. Min Range: -3.5573

Max Act: 2.6367. Min Act: -0.0000

Data Index: 292881 (C4 (Web Text))

Max Activating Token Index: 602

Text #5

Max Range: 3.5573. Min Range: -3.5573

Max Act: 2.4326. Min Act: -0.0000

Data Index: 1288600 (C4 (Web Text))

Max Activating Token Index: 360

Text #6

Max Range: 3.5573. Min Range: -3.5573

Max Act: 2.5407. Min Act: -0.0000

Data Index: 489238 (C4 (Web Text))

Max Activating Token Index: 197

Text #7

Max Range: 3.5573. Min Range: -3.5573

Max Act: 2.4982. Min Act: -0.0000

Data Index: 561007 (C4 (Web Text))

Max Activating Token Index: 236

Text #8

Max Range: 3.5573. Min Range: -3.5573

Max Act: 2.4056. Min Act: -0.0000

Data Index: 477571 (C4 (Web Text))

Max Activating Token Index: 153

Text #9

Max Range: 3.5573. Min Range: -3.5573

Max Act: 2.3013. Min Act: -0.0000

Data Index: 1253786 (C4 (Web Text))

Max Activating Token Index: 57

Text #10

Max Range: 3.5573. Min Range: -3.5573

Max Act: 2.1936. Min Act: -0.0000

Data Index: 1231215 (C4 (Web Text))

Max Activating Token Index: 185

Text #11

Max Range: 3.5573. Min Range: -3.5573

Max Act: 2.2068. Min Act: -0.0000

Data Index: 635376 (C4 (Web Text))

Max Activating Token Index: 811

Text #12

Max Range: 3.5573. Min Range: -3.5573

Max Act: 2.1674. Min Act: -0.0000

Data Index: 1122691 (C4 (Web Text))

Max Activating Token Index: 310

Text #13

Max Range: 3.5573. Min Range: -3.5573

Max Act: 1.8268. Min Act: -0.0000

Data Index: 970632 (C4 (Web Text))

Max Activating Token Index: 937

Text #14

Max Range: 3.5573. Min Range: -3.5573

Max Act: 1.9039. Min Act: -0.0000

Data Index: 932220 (C4 (Web Text))

Max Activating Token Index: 1001

Text #15

Max Range: 3.5573. Min Range: -3.5573

Max Act: 1.9707. Min Act: -0.0000

Data Index: 602904 (C4 (Web Text))

Max Activating Token Index: 753

Text #16

Max Range: 3.5573. Min Range: -3.5573

Max Act: 1.8156. Min Act: -0.0000

Data Index: 1240567 (C4 (Web Text))

Max Activating Token Index: 738

Text #17

Max Range: 3.5573. Min Range: -3.5573

Max Act: 1.9391. Min Act: -0.0000

Data Index: 174217 (C4 (Web Text))

Max Activating Token Index: 76

Text #18

Max Range: 3.5573. Min Range: -3.5573

Max Act: 1.9042. Min Act: -0.0000

Data Index: 495384 (C4 (Web Text))

Max Activating Token Index: 947

Text #19

Max Range: 3.5573. Min Range: -3.5573

Max Act: 1.9300. Min Act: -0.0000

Data Index: 42323 (C4 (Web Text))

Max Activating Token Index: 942