Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 216 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.3305. Min Range: -0.3305

Max Act: 0.3305. Min Act: -0.0000

Data Index: 785884 (C4 (Web Text))

Max Activating Token Index: 325

Text #1

Max Range: 0.3305. Min Range: -0.3305

Max Act: 0.3282. Min Act: -0.0000

Data Index: 1001335 (C4 (Web Text))

Max Activating Token Index: 294

Text #2

Max Range: 0.3305. Min Range: -0.3305

Max Act: 0.2934. Min Act: -0.0000

Data Index: 49371 (C4 (Web Text))

Max Activating Token Index: 918

Text #3

Max Range: 0.3305. Min Range: -0.3305

Max Act: 0.3173. Min Act: -0.0000

Data Index: 1344972 (C4 (Web Text))

Max Activating Token Index: 966

Text #4

Max Range: 0.3305. Min Range: -0.3305

Max Act: 0.3175. Min Act: -0.0000

Data Index: 40041 (C4 (Web Text))

Max Activating Token Index: 715

Text #5

Max Range: 0.3305. Min Range: -0.3305

Max Act: 0.2928. Min Act: -0.0000

Data Index: 458928 (C4 (Web Text))

Max Activating Token Index: 576

Text #6

Max Range: 0.3305. Min Range: -0.3305

Max Act: 0.3049. Min Act: -0.0000

Data Index: 895181 (C4 (Web Text))

Max Activating Token Index: 258

Text #7

Max Range: 0.3305. Min Range: -0.3305

Max Act: 0.2997. Min Act: -0.0000

Data Index: 1236445 (C4 (Web Text))

Max Activating Token Index: 949

Text #8

Max Range: 0.3305. Min Range: -0.3305

Max Act: 0.3001. Min Act: -0.0000

Data Index: 1224560 (C4 (Web Text))

Max Activating Token Index: 816

Text #9

Max Range: 0.3305. Min Range: -0.3305

Max Act: 0.2963. Min Act: -0.0000

Data Index: 323485 (C4 (Web Text))

Max Activating Token Index: 343

Text #10

Max Range: 0.3305. Min Range: -0.3305

Max Act: 0.3136. Min Act: -0.0000

Data Index: 299180 (C4 (Web Text))

Max Activating Token Index: 791

Text #11

Max Range: 0.3305. Min Range: -0.3305

Max Act: 0.2972. Min Act: -0.0000

Data Index: 172510 (C4 (Web Text))

Max Activating Token Index: 586

Text #12

Max Range: 0.3305. Min Range: -0.3305

Max Act: 0.2952. Min Act: -0.0000

Data Index: 682818 (C4 (Web Text))

Max Activating Token Index: 485

Text #13

Max Range: 0.3305. Min Range: -0.3305

Max Act: 0.3025. Min Act: -0.0000

Data Index: 932947 (C4 (Web Text))

Max Activating Token Index: 436

Text #14

Max Range: 0.3305. Min Range: -0.3305

Max Act: 0.2670. Min Act: -0.0000

Data Index: 603054 (C4 (Web Text))

Max Activating Token Index: 577

Text #15

Max Range: 0.3305. Min Range: -0.3305

Max Act: 0.2846. Min Act: -0.0000

Data Index: 443445 (C4 (Web Text))

Max Activating Token Index: 382

Text #16

Max Range: 0.3305. Min Range: -0.3305

Max Act: 0.2593. Min Act: -0.0000

Data Index: 1122858 (C4 (Web Text))

Max Activating Token Index: 596

Text #17

Max Range: 0.3305. Min Range: -0.3305

Max Act: 0.2876. Min Act: -0.0000

Data Index: 357226 (C4 (Web Text))

Max Activating Token Index: 392

Text #18

Max Range: 0.3305. Min Range: -0.3305

Max Act: 0.2963. Min Act: -0.0000

Data Index: 388679 (C4 (Web Text))

Max Activating Token Index: 357

Text #19

Max Range: 0.3305. Min Range: -0.3305

Max Act: 0.2837. Min Act: -0.0000

Data Index: 747641 (C4 (Web Text))

Max Activating Token Index: 523