Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1695 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.3955. Min Range: -0.3955

Max Act: 0.3955. Min Act: -0.0000

Data Index: 1610706 (Python Code)

Max Activating Token Index: 231

Text #1

Max Range: 0.3955. Min Range: -0.3955

Max Act: 0.3516. Min Act: -0.0000

Data Index: 218286 (C4 (Web Text))

Max Activating Token Index: 754

Text #2

Max Range: 0.3955. Min Range: -0.3955

Max Act: 0.3070. Min Act: -0.0000

Data Index: 1006759 (C4 (Web Text))

Max Activating Token Index: 766

Text #3

Max Range: 0.3955. Min Range: -0.3955

Max Act: 0.3464. Min Act: -0.0000

Data Index: 1537857 (Python Code)

Max Activating Token Index: 299

Text #4

Max Range: 0.3955. Min Range: -0.3955

Max Act: 0.3407. Min Act: -0.0000

Data Index: 1462692 (Python Code)

Max Activating Token Index: 232

Text #5

Max Range: 0.3955. Min Range: -0.3955

Max Act: 0.3487. Min Act: -0.0000

Data Index: 1134167 (C4 (Web Text))

Max Activating Token Index: 940

Text #6

Max Range: 0.3955. Min Range: -0.3955

Max Act: 0.3080. Min Act: -0.0000

Data Index: 1548050 (Python Code)

Max Activating Token Index: 255

Text #7

Max Range: 0.3955. Min Range: -0.3955

Max Act: 0.3146. Min Act: -0.0000

Data Index: 1617767 (Python Code)

Max Activating Token Index: 248

Text #8

Max Range: 0.3955. Min Range: -0.3955

Max Act: 0.3230. Min Act: -0.0000

Data Index: 1577313 (Python Code)

Max Activating Token Index: 359

Text #9

Max Range: 0.3955. Min Range: -0.3955

Max Act: 0.3131. Min Act: -0.0000

Data Index: 880361 (C4 (Web Text))

Max Activating Token Index: 943

Text #10

Max Range: 0.3955. Min Range: -0.3955

Max Act: 0.3154. Min Act: -0.0000

Data Index: 43122 (C4 (Web Text))

Max Activating Token Index: 428

Text #11

Max Range: 0.3955. Min Range: -0.3955

Max Act: 0.3232. Min Act: -0.0000

Data Index: 1544225 (Python Code)

Max Activating Token Index: 110

Text #12

Max Range: 0.3955. Min Range: -0.3955

Max Act: 0.3435. Min Act: -0.0000

Data Index: 1350903 (C4 (Web Text))

Max Activating Token Index: 720

Text #13

Max Range: 0.3955. Min Range: -0.3955

Max Act: 0.3315. Min Act: -0.0000

Data Index: 109097 (C4 (Web Text))

Max Activating Token Index: 777

Text #14

Max Range: 0.3955. Min Range: -0.3955

Max Act: 0.3050. Min Act: -0.0000

Data Index: 451319 (C4 (Web Text))

Max Activating Token Index: 605

Text #15

Max Range: 0.3955. Min Range: -0.3955

Max Act: 0.3153. Min Act: -0.0000

Data Index: 1652721 (Python Code)

Max Activating Token Index: 252

Text #16

Max Range: 0.3955. Min Range: -0.3955

Max Act: 0.3091. Min Act: -0.0000

Data Index: 362976 (C4 (Web Text))

Max Activating Token Index: 23

Text #17

Max Range: 0.3955. Min Range: -0.3955

Max Act: 0.3070. Min Act: -0.0000

Data Index: 406926 (C4 (Web Text))

Max Activating Token Index: 675

Text #18

Max Range: 0.3955. Min Range: -0.3955

Max Act: 0.2734. Min Act: -0.0000

Data Index: 606629 (C4 (Web Text))

Max Activating Token Index: 250

Text #19

Max Range: 0.3955. Min Range: -0.3955

Max Act: 0.2766. Min Act: -0.0000

Data Index: 944134 (C4 (Web Text))

Max Activating Token Index: 415