Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 759 in Layer 11

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 0.8666. Min Range: -0.8666

Max Act: 0.8666. Min Act: -0.0000

Data Index: 571579 (C4 (Web Text))

Max Activating Token Index: 759

Text #1

Max Range: 0.8666. Min Range: -0.8666

Max Act: 0.5606. Min Act: -0.0000

Data Index: 1191838 (C4 (Web Text))

Max Activating Token Index: 380

Text #2

Max Range: 0.8666. Min Range: -0.8666

Max Act: 0.5242. Min Act: -0.0000

Data Index: 522558 (C4 (Web Text))

Max Activating Token Index: 387

Text #3

Max Range: 0.8666. Min Range: -0.8666

Max Act: 0.5073. Min Act: -0.0000

Data Index: 1279100 (C4 (Web Text))

Max Activating Token Index: 340

Text #4

Max Range: 0.8666. Min Range: -0.8666

Max Act: 0.6114. Min Act: -0.0000

Data Index: 122316 (C4 (Web Text))

Max Activating Token Index: 309

Text #5

Max Range: 0.8666. Min Range: -0.8666

Max Act: 0.5035. Min Act: -0.0000

Data Index: 6344 (C4 (Web Text))

Max Activating Token Index: 672

Text #6

Max Range: 0.8666. Min Range: -0.8666

Max Act: 0.4689. Min Act: -0.0000

Data Index: 758823 (C4 (Web Text))

Max Activating Token Index: 614

Text #7

Max Range: 0.8666. Min Range: -0.8666

Max Act: 0.4735. Min Act: -0.0000

Data Index: 33530 (C4 (Web Text))

Max Activating Token Index: 352

Text #8

Max Range: 0.8666. Min Range: -0.8666

Max Act: 0.4616. Min Act: 0.0000

Data Index: 718751 (C4 (Web Text))

Max Activating Token Index: 450

Text #9

Max Range: 0.8666. Min Range: -0.8666

Max Act: 0.5115. Min Act: -0.0000

Data Index: 18770 (C4 (Web Text))

Max Activating Token Index: 953

Text #10

Max Range: 0.8666. Min Range: -0.8666

Max Act: 0.4831. Min Act: -0.0000

Data Index: 892160 (C4 (Web Text))

Max Activating Token Index: 611

Text #11

Max Range: 0.8666. Min Range: -0.8666

Max Act: 0.4204. Min Act: -0.0000

Data Index: 324521 (C4 (Web Text))

Max Activating Token Index: 378

Text #12

Max Range: 0.8666. Min Range: -0.8666

Max Act: 0.4394. Min Act: -0.0000

Data Index: 120092 (C4 (Web Text))

Max Activating Token Index: 507

Text #13

Max Range: 0.8666. Min Range: -0.8666

Max Act: 0.4353. Min Act: -0.0000

Data Index: 901777 (C4 (Web Text))

Max Activating Token Index: 806

Text #14

Max Range: 0.8666. Min Range: -0.8666

Max Act: 0.4176. Min Act: -0.0000

Data Index: 518341 (C4 (Web Text))

Max Activating Token Index: 197

Text #15

Max Range: 0.8666. Min Range: -0.8666

Max Act: 0.5052. Min Act: -0.0000

Data Index: 222408 (C4 (Web Text))

Max Activating Token Index: 254

Text #16

Max Range: 0.8666. Min Range: -0.8666

Max Act: 0.4385. Min Act: -0.0000

Data Index: 1359519 (C4 (Web Text))

Max Activating Token Index: 931

Text #17

Max Range: 0.8666. Min Range: -0.8666

Max Act: 0.3930. Min Act: 0.0001

Data Index: 997730 (C4 (Web Text))

Max Activating Token Index: 184

Text #18

Max Range: 0.8666. Min Range: -0.8666

Max Act: 0.4026. Min Act: -0.0000

Data Index: 1133943 (C4 (Web Text))

Max Activating Token Index: 115

Text #19

Max Range: 0.8666. Min Range: -0.8666

Max Act: 0.4295. Min Act: -0.0000

Data Index: 685325 (C4 (Web Text))

Max Activating Token Index: 448