Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4056 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 2.0766. Min Range: -2.0766

Max Act: 2.0766. Min Act: -0.0000

Data Index: 1463723 (Python Code)

Max Activating Token Index: 41

Text #1

Max Range: 2.0766. Min Range: -2.0766

Max Act: 1.7916. Min Act: -0.0000

Data Index: 1544357 (Python Code)

Max Activating Token Index: 276

Text #2

Max Range: 2.0766. Min Range: -2.0766

Max Act: 1.6868. Min Act: -0.0000

Data Index: 1612632 (Python Code)

Max Activating Token Index: 317

Text #3

Max Range: 2.0766. Min Range: -2.0766

Max Act: 1.6799. Min Act: -0.0000

Data Index: 1494397 (Python Code)

Max Activating Token Index: 608

Text #4

Max Range: 2.0766. Min Range: -2.0766

Max Act: 1.6896. Min Act: -0.0000

Data Index: 881491 (C4 (Web Text))

Max Activating Token Index: 893

Text #5

Max Range: 2.0766. Min Range: -2.0766

Max Act: 1.6869. Min Act: -0.0000

Data Index: 1583190 (Python Code)

Max Activating Token Index: 37

Text #6

Max Range: 2.0766. Min Range: -2.0766

Max Act: 1.6507. Min Act: -0.0000

Data Index: 813369 (C4 (Web Text))

Max Activating Token Index: 165

Text #7

Max Range: 2.0766. Min Range: -2.0766

Max Act: 1.4737. Min Act: -0.0000

Data Index: 643220 (C4 (Web Text))

Max Activating Token Index: 318

Text #8

Max Range: 2.0766. Min Range: -2.0766

Max Act: 1.5191. Min Act: -0.0000

Data Index: 1424844 (Python Code)

Max Activating Token Index: 587

Text #9

Max Range: 2.0766. Min Range: -2.0766

Max Act: 1.6137. Min Act: -0.0000

Data Index: 469964 (C4 (Web Text))

Max Activating Token Index: 1007

Text #10

Max Range: 2.0766. Min Range: -2.0766

Max Act: 1.5283. Min Act: -0.0000

Data Index: 280284 (C4 (Web Text))

Max Activating Token Index: 433

Text #11

Max Range: 2.0766. Min Range: -2.0766

Max Act: 1.3610. Min Act: -0.0000

Data Index: 472976 (C4 (Web Text))

Max Activating Token Index: 508

Text #12

Max Range: 2.0766. Min Range: -2.0766

Max Act: 1.4522. Min Act: -0.0000

Data Index: 683127 (C4 (Web Text))

Max Activating Token Index: 895

Text #13

Max Range: 2.0766. Min Range: -2.0766

Max Act: 1.3661. Min Act: -0.0000

Data Index: 107422 (C4 (Web Text))

Max Activating Token Index: 493

Text #14

Max Range: 2.0766. Min Range: -2.0766

Max Act: 1.4471. Min Act: -0.0000

Data Index: 1546308 (Python Code)

Max Activating Token Index: 831

Text #15

Max Range: 2.0766. Min Range: -2.0766

Max Act: 1.4572. Min Act: -0.0000

Data Index: 442405 (C4 (Web Text))

Max Activating Token Index: 6

Text #16

Max Range: 2.0766. Min Range: -2.0766

Max Act: 1.3817. Min Act: -0.0000

Data Index: 501870 (C4 (Web Text))

Max Activating Token Index: 814

Text #17

Max Range: 2.0766. Min Range: -2.0766

Max Act: 1.4870. Min Act: -0.0000

Data Index: 971563 (C4 (Web Text))

Max Activating Token Index: 15

Text #18

Max Range: 2.0766. Min Range: -2.0766

Max Act: 1.4449. Min Act: -0.0000

Data Index: 1051625 (C4 (Web Text))

Max Activating Token Index: 509

Text #19

Max Range: 2.0766. Min Range: -2.0766

Max Act: 1.4127. Min Act: -0.0000

Data Index: 525533 (C4 (Web Text))

Max Activating Token Index: 65