Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3029 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 0.3717. Min Range: -0.3717

Max Act: 0.3671. Min Act: -0.0000

Data Index: 912069 (C4 (Web Text))

Max Activating Token Index: 356

Text #1

Max Range: 0.3717. Min Range: -0.3717

Max Act: 0.3717. Min Act: -0.0000

Data Index: 749678 (C4 (Web Text))

Max Activating Token Index: 635

Text #2

Max Range: 0.3717. Min Range: -0.3717

Max Act: 0.3319. Min Act: -0.0000

Data Index: 1233572 (C4 (Web Text))

Max Activating Token Index: 404

Text #3

Max Range: 0.3717. Min Range: -0.3717

Max Act: 0.3572. Min Act: -0.0000

Data Index: 536203 (C4 (Web Text))

Max Activating Token Index: 858

Text #4

Max Range: 0.3717. Min Range: -0.3717

Max Act: 0.3409. Min Act: -0.0000

Data Index: 999012 (C4 (Web Text))

Max Activating Token Index: 499

Text #5

Max Range: 0.3717. Min Range: -0.3717

Max Act: 0.3632. Min Act: -0.0000

Data Index: 108615 (C4 (Web Text))

Max Activating Token Index: 907

Text #6

Max Range: 0.3717. Min Range: -0.3717

Max Act: 0.3120. Min Act: -0.0000

Data Index: 724459 (C4 (Web Text))

Max Activating Token Index: 965

Text #7

Max Range: 0.3717. Min Range: -0.3717

Max Act: 0.3099. Min Act: -0.0000

Data Index: 871088 (C4 (Web Text))

Max Activating Token Index: 838

Text #8

Max Range: 0.3717. Min Range: -0.3717

Max Act: 0.2762. Min Act: -0.0000

Data Index: 797146 (C4 (Web Text))

Max Activating Token Index: 243

Text #9

Max Range: 0.3717. Min Range: -0.3717

Max Act: 0.2924. Min Act: -0.0000

Data Index: 562488 (C4 (Web Text))

Max Activating Token Index: 494

Text #10

Max Range: 0.3717. Min Range: -0.3717

Max Act: 0.2626. Min Act: -0.0000

Data Index: 662665 (C4 (Web Text))

Max Activating Token Index: 179

Text #11

Max Range: 0.3717. Min Range: -0.3717

Max Act: 0.2851. Min Act: -0.0000

Data Index: 1307580 (C4 (Web Text))

Max Activating Token Index: 513

Text #12

Max Range: 0.3717. Min Range: -0.3717

Max Act: 0.2657. Min Act: -0.0000

Data Index: 373564 (C4 (Web Text))

Max Activating Token Index: 985

Text #13

Max Range: 0.3717. Min Range: -0.3717

Max Act: 0.2693. Min Act: -0.0000

Data Index: 950548 (C4 (Web Text))

Max Activating Token Index: 692

Text #14

Max Range: 0.3717. Min Range: -0.3717

Max Act: 0.2705. Min Act: -0.0000

Data Index: 205834 (C4 (Web Text))

Max Activating Token Index: 503

Text #15

Max Range: 0.3717. Min Range: -0.3717

Max Act: 0.2494. Min Act: -0.0000

Data Index: 264587 (C4 (Web Text))

Max Activating Token Index: 485

Text #16

Max Range: 0.3717. Min Range: -0.3717

Max Act: 0.2716. Min Act: -0.0000

Data Index: 1094257 (C4 (Web Text))

Max Activating Token Index: 985

Text #17

Max Range: 0.3717. Min Range: -0.3717

Max Act: 0.2302. Min Act: -0.0000

Data Index: 1057107 (C4 (Web Text))

Max Activating Token Index: 748

Text #18

Max Range: 0.3717. Min Range: -0.3717

Max Act: 0.2488. Min Act: -0.0000

Data Index: 432335 (C4 (Web Text))

Max Activating Token Index: 883

Text #19

Max Range: 0.3717. Min Range: -0.3717

Max Act: 0.2383. Min Act: -0.0000

Data Index: 383707 (C4 (Web Text))

Max Activating Token Index: 946