Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1864 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 3.3739. Min Range: -3.3739

Max Act: 3.3739. Min Act: -0.0000

Data Index: 928487 (C4 (Web Text))

Max Activating Token Index: 116

Text #1

Max Range: 3.3739. Min Range: -3.3739

Max Act: 3.1942. Min Act: -0.0000

Data Index: 356327 (C4 (Web Text))

Max Activating Token Index: 208

Text #2

Max Range: 3.3739. Min Range: -3.3739

Max Act: 2.9732. Min Act: -0.0000

Data Index: 276144 (C4 (Web Text))

Max Activating Token Index: 846

Text #3

Max Range: 3.3739. Min Range: -3.3739

Max Act: 2.6830. Min Act: -0.0000

Data Index: 219505 (C4 (Web Text))

Max Activating Token Index: 770

Text #4

Max Range: 3.3739. Min Range: -3.3739

Max Act: 2.6440. Min Act: -0.0000

Data Index: 976291 (C4 (Web Text))

Max Activating Token Index: 182

Text #5

Max Range: 3.3739. Min Range: -3.3739

Max Act: 2.6050. Min Act: -0.0000

Data Index: 1132615 (C4 (Web Text))

Max Activating Token Index: 449

Text #6

Max Range: 3.3739. Min Range: -3.3739

Max Act: 2.4855. Min Act: -0.0000

Data Index: 441189 (C4 (Web Text))

Max Activating Token Index: 30

Text #7

Max Range: 3.3739. Min Range: -3.3739

Max Act: 2.6913. Min Act: -0.0000

Data Index: 975537 (C4 (Web Text))

Max Activating Token Index: 577

Text #8

Max Range: 3.3739. Min Range: -3.3739

Max Act: 2.7235. Min Act: -0.0000

Data Index: 831957 (C4 (Web Text))

Max Activating Token Index: 88

Text #9

Max Range: 3.3739. Min Range: -3.3739

Max Act: 2.6037. Min Act: -0.0000

Data Index: 1249244 (C4 (Web Text))

Max Activating Token Index: 250

Text #10

Max Range: 3.3739. Min Range: -3.3739

Max Act: 2.4602. Min Act: -0.0000

Data Index: 760464 (C4 (Web Text))

Max Activating Token Index: 166

Text #11

Max Range: 3.3739. Min Range: -3.3739

Max Act: 2.4183. Min Act: -0.0000

Data Index: 1374309 (Python Code)

Max Activating Token Index: 165

Text #12

Max Range: 3.3739. Min Range: -3.3739

Max Act: 2.1141. Min Act: -0.0000

Data Index: 1270679 (C4 (Web Text))

Max Activating Token Index: 752

Text #13

Max Range: 3.3739. Min Range: -3.3739

Max Act: 2.4671. Min Act: -0.0000

Data Index: 736482 (C4 (Web Text))

Max Activating Token Index: 58

Text #14

Max Range: 3.3739. Min Range: -3.3739

Max Act: 2.5543. Min Act: -0.0000

Data Index: 965720 (C4 (Web Text))

Max Activating Token Index: 64

Text #15

Max Range: 3.3739. Min Range: -3.3739

Max Act: 2.4313. Min Act: -0.0000

Data Index: 296735 (C4 (Web Text))

Max Activating Token Index: 151

Text #16

Max Range: 3.3739. Min Range: -3.3739

Max Act: 2.4190. Min Act: -0.0000

Data Index: 1056693 (C4 (Web Text))

Max Activating Token Index: 133

Text #17

Max Range: 3.3739. Min Range: -3.3739

Max Act: 2.4150. Min Act: -0.0000

Data Index: 693988 (C4 (Web Text))

Max Activating Token Index: 76

Text #18

Max Range: 3.3739. Min Range: -3.3739

Max Act: 2.3987. Min Act: -0.0000

Data Index: 827923 (C4 (Web Text))

Max Activating Token Index: 707

Text #19

Max Range: 3.3739. Min Range: -3.3739

Max Act: 2.3838. Min Act: -0.0000

Data Index: 142206 (C4 (Web Text))

Max Activating Token Index: 180