Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2672 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 4.3853. Min Range: -4.3853

Max Act: 4.3853. Min Act: -0.0000

Data Index: 152526 (C4 (Web Text))

Max Activating Token Index: 209

Text #1

Max Range: 4.3853. Min Range: -4.3853

Max Act: 3.6643. Min Act: -0.0000

Data Index: 1272809 (C4 (Web Text))

Max Activating Token Index: 404

Text #2

Max Range: 4.3853. Min Range: -4.3853

Max Act: 3.4715. Min Act: -0.0000

Data Index: 1226344 (C4 (Web Text))

Max Activating Token Index: 39

Text #3

Max Range: 4.3853. Min Range: -4.3853

Max Act: 3.3368. Min Act: -0.0000

Data Index: 453571 (C4 (Web Text))

Max Activating Token Index: 489

Text #4

Max Range: 4.3853. Min Range: -4.3853

Max Act: 2.9425. Min Act: -0.0000

Data Index: 1126257 (C4 (Web Text))

Max Activating Token Index: 705

Text #5

Max Range: 4.3853. Min Range: -4.3853

Max Act: 3.3722. Min Act: -0.0000

Data Index: 294515 (C4 (Web Text))

Max Activating Token Index: 655

Text #6

Max Range: 4.3853. Min Range: -4.3853

Max Act: 2.7620. Min Act: -0.0000

Data Index: 9261 (C4 (Web Text))

Max Activating Token Index: 186

Text #7

Max Range: 4.3853. Min Range: -4.3853

Max Act: 2.6559. Min Act: -0.0000

Data Index: 209508 (C4 (Web Text))

Max Activating Token Index: 47

Text #8

Max Range: 4.3853. Min Range: -4.3853

Max Act: 2.9450. Min Act: -0.0000

Data Index: 672533 (C4 (Web Text))

Max Activating Token Index: 104

Text #9

Max Range: 4.3853. Min Range: -4.3853

Max Act: 2.7919. Min Act: -0.0000

Data Index: 82065 (C4 (Web Text))

Max Activating Token Index: 392

Text #10

Max Range: 4.3853. Min Range: -4.3853

Max Act: 2.7975. Min Act: -0.0000

Data Index: 1347682 (C4 (Web Text))

Max Activating Token Index: 354

Text #11

Max Range: 4.3853. Min Range: -4.3853

Max Act: 2.7843. Min Act: -0.0000

Data Index: 655509 (C4 (Web Text))

Max Activating Token Index: 62

Text #12

Max Range: 4.3853. Min Range: -4.3853

Max Act: 2.7113. Min Act: -0.0000

Data Index: 464620 (C4 (Web Text))

Max Activating Token Index: 156

Text #13

Max Range: 4.3853. Min Range: -4.3853

Max Act: 2.6415. Min Act: -0.0000

Data Index: 295069 (C4 (Web Text))

Max Activating Token Index: 841

Text #14

Max Range: 4.3853. Min Range: -4.3853

Max Act: 2.4816. Min Act: -0.0000

Data Index: 1359024 (C4 (Web Text))

Max Activating Token Index: 484

Text #15

Max Range: 4.3853. Min Range: -4.3853

Max Act: 2.5781. Min Act: -0.0000

Data Index: 574116 (C4 (Web Text))

Max Activating Token Index: 958

Text #16

Max Range: 4.3853. Min Range: -4.3853

Max Act: 2.5368. Min Act: -0.0000

Data Index: 1293723 (C4 (Web Text))

Max Activating Token Index: 799

Text #17

Max Range: 4.3853. Min Range: -4.3853

Max Act: 2.4943. Min Act: -0.0000

Data Index: 617801 (C4 (Web Text))

Max Activating Token Index: 30

Text #18

Max Range: 4.3853. Min Range: -4.3853

Max Act: 2.5103. Min Act: -0.0000

Data Index: 1265018 (C4 (Web Text))

Max Activating Token Index: 749

Text #19

Max Range: 4.3853. Min Range: -4.3853

Max Act: 2.6386. Min Act: -0.0000

Data Index: 1075064 (C4 (Web Text))

Max Activating Token Index: 377