Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2698 in Layer 3

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.5849. Min Range: -0.5849

Max Act: 0.4964. Min Act: -0.0001

Data Index: 1162747 (C4 (Web Text))

Max Activating Token Index: 684

Text #1

Max Range: 0.5849. Min Range: -0.5849

Max Act: 0.5849. Min Act: -0.0001

Data Index: 1081302 (C4 (Web Text))

Max Activating Token Index: 32

Text #2

Max Range: 0.5849. Min Range: -0.5849

Max Act: 0.5311. Min Act: -0.0001

Data Index: 19926 (C4 (Web Text))

Max Activating Token Index: 207

Text #3

Max Range: 0.5849. Min Range: -0.5849

Max Act: 0.5452. Min Act: -0.0001

Data Index: 813765 (C4 (Web Text))

Max Activating Token Index: 95

Text #4

Max Range: 0.5849. Min Range: -0.5849

Max Act: 0.4409. Min Act: -0.0001

Data Index: 225478 (C4 (Web Text))

Max Activating Token Index: 36

Text #5

Max Range: 0.5849. Min Range: -0.5849

Max Act: 0.4944. Min Act: -0.0001

Data Index: 955323 (C4 (Web Text))

Max Activating Token Index: 543

Text #6

Max Range: 0.5849. Min Range: -0.5849

Max Act: 0.5601. Min Act: -0.0001

Data Index: 1089798 (C4 (Web Text))

Max Activating Token Index: 893

Text #7

Max Range: 0.5849. Min Range: -0.5849

Max Act: 0.5084. Min Act: -0.0001

Data Index: 574101 (C4 (Web Text))

Max Activating Token Index: 146

Text #8

Max Range: 0.5849. Min Range: -0.5849

Max Act: 0.4097. Min Act: -0.0001

Data Index: 142858 (C4 (Web Text))

Max Activating Token Index: 56

Text #9

Max Range: 0.5849. Min Range: -0.5849

Max Act: 0.4689. Min Act: -0.0001

Data Index: 720668 (C4 (Web Text))

Max Activating Token Index: 20

Text #10

Max Range: 0.5849. Min Range: -0.5849

Max Act: 0.4403. Min Act: -0.0001

Data Index: 382886 (C4 (Web Text))

Max Activating Token Index: 120

Text #11

Max Range: 0.5849. Min Range: -0.5849

Max Act: 0.4478. Min Act: -0.0001

Data Index: 1102792 (C4 (Web Text))

Max Activating Token Index: 16

Text #12

Max Range: 0.5849. Min Range: -0.5849

Max Act: 0.4832. Min Act: -0.0001

Data Index: 4704 (C4 (Web Text))

Max Activating Token Index: 494

Text #13

Max Range: 0.5849. Min Range: -0.5849

Max Act: 0.4560. Min Act: -0.0001

Data Index: 1089151 (C4 (Web Text))

Max Activating Token Index: 522

Text #14

Max Range: 0.5849. Min Range: -0.5849

Max Act: 0.4701. Min Act: -0.0001

Data Index: 505892 (C4 (Web Text))

Max Activating Token Index: 756

Text #15

Max Range: 0.5849. Min Range: -0.5849

Max Act: 0.4655. Min Act: -0.0001

Data Index: 881137 (C4 (Web Text))

Max Activating Token Index: 373

Text #16

Max Range: 0.5849. Min Range: -0.5849

Max Act: 0.4961. Min Act: -0.0001

Data Index: 77946 (C4 (Web Text))

Max Activating Token Index: 494

Text #17

Max Range: 0.5849. Min Range: -0.5849

Max Act: 0.4080. Min Act: -0.0001

Data Index: 810706 (C4 (Web Text))

Max Activating Token Index: 55

Text #18

Max Range: 0.5849. Min Range: -0.5849

Max Act: 0.4655. Min Act: -0.0001

Data Index: 242044 (C4 (Web Text))

Max Activating Token Index: 864

Text #19

Max Range: 0.5849. Min Range: -0.5849

Max Act: 0.4914. Min Act: -0.0001

Data Index: 1068680 (C4 (Web Text))

Max Activating Token Index: 57