Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3969 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.9166. Min Range: -0.9166

Max Act: 0.8541. Min Act: -0.0000

Data Index: 1499445 (Python Code)

Max Activating Token Index: 582

Text #1

Max Range: 0.9166. Min Range: -0.9166

Max Act: 0.9166. Min Act: -0.0000

Data Index: 1425431 (Python Code)

Max Activating Token Index: 708

Text #2

Max Range: 0.9166. Min Range: -0.9166

Max Act: 0.8305. Min Act: -0.0000

Data Index: 1377229 (Python Code)

Max Activating Token Index: 591

Text #3

Max Range: 0.9166. Min Range: -0.9166

Max Act: 0.8459. Min Act: -0.0000

Data Index: 1533178 (Python Code)

Max Activating Token Index: 437

Text #4

Max Range: 0.9166. Min Range: -0.9166

Max Act: 0.7509. Min Act: -0.0000

Data Index: 1609789 (Python Code)

Max Activating Token Index: 919

Text #5

Max Range: 0.9166. Min Range: -0.9166

Max Act: 0.7553. Min Act: -0.0000

Data Index: 1615095 (Python Code)

Max Activating Token Index: 189

Text #6

Max Range: 0.9166. Min Range: -0.9166

Max Act: 0.7192. Min Act: -0.0000

Data Index: 1266724 (C4 (Web Text))

Max Activating Token Index: 526

Text #7

Max Range: 0.9166. Min Range: -0.9166

Max Act: 0.7887. Min Act: -0.0000

Data Index: 1550664 (Python Code)

Max Activating Token Index: 679

Text #8

Max Range: 0.9166. Min Range: -0.9166

Max Act: 0.7602. Min Act: -0.0000

Data Index: 1577354 (Python Code)

Max Activating Token Index: 299

Text #9

Max Range: 0.9166. Min Range: -0.9166

Max Act: 0.6734. Min Act: -0.0000

Data Index: 446625 (C4 (Web Text))

Max Activating Token Index: 899

Text #10

Max Range: 0.9166. Min Range: -0.9166

Max Act: 0.6339. Min Act: -0.0000

Data Index: 1588955 (Python Code)

Max Activating Token Index: 518

Text #11

Max Range: 0.9166. Min Range: -0.9166

Max Act: 0.6400. Min Act: -0.0000

Data Index: 1455359 (Python Code)

Max Activating Token Index: 275

Text #12

Max Range: 0.9166. Min Range: -0.9166

Max Act: 0.6835. Min Act: -0.0000

Data Index: 1628222 (Python Code)

Max Activating Token Index: 557

Text #13

Max Range: 0.9166. Min Range: -0.9166

Max Act: 0.7473. Min Act: -0.0000

Data Index: 1638649 (Python Code)

Max Activating Token Index: 343

Text #14

Max Range: 0.9166. Min Range: -0.9166

Max Act: 0.5260. Min Act: -0.0000

Data Index: 1408928 (Python Code)

Max Activating Token Index: 173

Text #15

Max Range: 0.9166. Min Range: -0.9166

Max Act: 0.6118. Min Act: -0.0000

Data Index: 1484736 (Python Code)

Max Activating Token Index: 880

Text #16

Max Range: 0.9166. Min Range: -0.9166

Max Act: 0.6713. Min Act: -0.0000

Data Index: 811486 (C4 (Web Text))

Max Activating Token Index: 903

Text #17

Max Range: 0.9166. Min Range: -0.9166

Max Act: 0.6275. Min Act: -0.0000

Data Index: 1423049 (Python Code)

Max Activating Token Index: 492

Text #18

Max Range: 0.9166. Min Range: -0.9166

Max Act: 0.6024. Min Act: -0.0000

Data Index: 1376882 (Python Code)

Max Activating Token Index: 717

Text #19

Max Range: 0.9166. Min Range: -0.9166

Max Act: 0.5691. Min Act: -0.0000

Data Index: 1532634 (Python Code)

Max Activating Token Index: 731