Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1410 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.0919. Min Range: -1.0919

Max Act: 1.0919. Min Act: -0.0000

Data Index: 1034616 (C4 (Web Text))

Max Activating Token Index: 420

Text #1

Max Range: 1.0919. Min Range: -1.0919

Max Act: 1.0141. Min Act: -0.0000

Data Index: 876347 (C4 (Web Text))

Max Activating Token Index: 227

Text #2

Max Range: 1.0919. Min Range: -1.0919

Max Act: 0.9718. Min Act: -0.0000

Data Index: 432162 (C4 (Web Text))

Max Activating Token Index: 696

Text #3

Max Range: 1.0919. Min Range: -1.0919

Max Act: 0.8858. Min Act: -0.0000

Data Index: 1204153 (C4 (Web Text))

Max Activating Token Index: 215

Text #4

Max Range: 1.0919. Min Range: -1.0919

Max Act: 0.9007. Min Act: -0.0000

Data Index: 1058422 (C4 (Web Text))

Max Activating Token Index: 205

Text #5

Max Range: 1.0919. Min Range: -1.0919

Max Act: 0.8770. Min Act: -0.0000

Data Index: 384308 (C4 (Web Text))

Max Activating Token Index: 261

Text #6

Max Range: 1.0919. Min Range: -1.0919

Max Act: 0.8854. Min Act: -0.0000

Data Index: 699565 (C4 (Web Text))

Max Activating Token Index: 285

Text #7

Max Range: 1.0919. Min Range: -1.0919

Max Act: 0.9071. Min Act: -0.0000

Data Index: 609068 (C4 (Web Text))

Max Activating Token Index: 329

Text #8

Max Range: 1.0919. Min Range: -1.0919

Max Act: 0.8744. Min Act: -0.0000

Data Index: 990718 (C4 (Web Text))

Max Activating Token Index: 793

Text #9

Max Range: 1.0919. Min Range: -1.0919

Max Act: 0.8608. Min Act: -0.0000

Data Index: 481029 (C4 (Web Text))

Max Activating Token Index: 879

Text #10

Max Range: 1.0919. Min Range: -1.0919

Max Act: 0.8857. Min Act: -0.0000

Data Index: 597196 (C4 (Web Text))

Max Activating Token Index: 738

Text #11

Max Range: 1.0919. Min Range: -1.0919

Max Act: 0.8293. Min Act: -0.0000

Data Index: 919723 (C4 (Web Text))

Max Activating Token Index: 881

Text #12

Max Range: 1.0919. Min Range: -1.0919

Max Act: 0.7838. Min Act: -0.0000

Data Index: 1354127 (C4 (Web Text))

Max Activating Token Index: 316

Text #13

Max Range: 1.0919. Min Range: -1.0919

Max Act: 0.8552. Min Act: -0.0000

Data Index: 1340134 (C4 (Web Text))

Max Activating Token Index: 811

Text #14

Max Range: 1.0919. Min Range: -1.0919

Max Act: 0.8619. Min Act: -0.0000

Data Index: 405312 (C4 (Web Text))

Max Activating Token Index: 947

Text #15

Max Range: 1.0919. Min Range: -1.0919

Max Act: 0.8489. Min Act: -0.0000

Data Index: 111122 (C4 (Web Text))

Max Activating Token Index: 287

Text #16

Max Range: 1.0919. Min Range: -1.0919

Max Act: 0.8332. Min Act: -0.0000

Data Index: 768716 (C4 (Web Text))

Max Activating Token Index: 121

Text #17

Max Range: 1.0919. Min Range: -1.0919

Max Act: 0.8456. Min Act: -0.0000

Data Index: 1315529 (C4 (Web Text))

Max Activating Token Index: 559

Text #18

Max Range: 1.0919. Min Range: -1.0919

Max Act: 0.7772. Min Act: -0.0000

Data Index: 1087034 (C4 (Web Text))

Max Activating Token Index: 525

Text #19

Max Range: 1.0919. Min Range: -1.0919

Max Act: 0.8300. Min Act: -0.0000

Data Index: 42753 (C4 (Web Text))

Max Activating Token Index: 978