Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4513 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.0007. Min Range: -1.0007

Max Act: 0.9616. Min Act: -0.0000

Data Index: 1052322 (C4 (Web Text))

Max Activating Token Index: 443

Text #1

Max Range: 1.0007. Min Range: -1.0007

Max Act: 1.0007. Min Act: -0.0000

Data Index: 671193 (C4 (Web Text))

Max Activating Token Index: 269

Text #2

Max Range: 1.0007. Min Range: -1.0007

Max Act: 0.8352. Min Act: -0.0000

Data Index: 757123 (C4 (Web Text))

Max Activating Token Index: 735

Text #3

Max Range: 1.0007. Min Range: -1.0007

Max Act: 0.8907. Min Act: -0.0000

Data Index: 5839 (C4 (Web Text))

Max Activating Token Index: 204

Text #4

Max Range: 1.0007. Min Range: -1.0007

Max Act: 0.9034. Min Act: -0.0000

Data Index: 269058 (C4 (Web Text))

Max Activating Token Index: 522

Text #5

Max Range: 1.0007. Min Range: -1.0007

Max Act: 0.8003. Min Act: -0.0000

Data Index: 462874 (C4 (Web Text))

Max Activating Token Index: 644

Text #6

Max Range: 1.0007. Min Range: -1.0007

Max Act: 0.8184. Min Act: -0.0000

Data Index: 1086034 (C4 (Web Text))

Max Activating Token Index: 673

Text #7

Max Range: 1.0007. Min Range: -1.0007

Max Act: 0.9039. Min Act: -0.0000

Data Index: 941937 (C4 (Web Text))

Max Activating Token Index: 289

Text #8

Max Range: 1.0007. Min Range: -1.0007

Max Act: 0.8080. Min Act: -0.0000

Data Index: 587392 (C4 (Web Text))

Max Activating Token Index: 605

Text #9

Max Range: 1.0007. Min Range: -1.0007

Max Act: 0.8897. Min Act: -0.0000

Data Index: 53152 (C4 (Web Text))

Max Activating Token Index: 821

Text #10

Max Range: 1.0007. Min Range: -1.0007

Max Act: 0.8270. Min Act: -0.0000

Data Index: 705197 (C4 (Web Text))

Max Activating Token Index: 555

Text #11

Max Range: 1.0007. Min Range: -1.0007

Max Act: 0.8737. Min Act: -0.0000

Data Index: 1346297 (C4 (Web Text))

Max Activating Token Index: 304

Text #12

Max Range: 1.0007. Min Range: -1.0007

Max Act: 0.8459. Min Act: -0.0000

Data Index: 1004508 (C4 (Web Text))

Max Activating Token Index: 907

Text #13

Max Range: 1.0007. Min Range: -1.0007

Max Act: 0.8174. Min Act: -0.0000

Data Index: 472696 (C4 (Web Text))

Max Activating Token Index: 876

Text #14

Max Range: 1.0007. Min Range: -1.0007

Max Act: 0.8411. Min Act: -0.0000

Data Index: 349562 (C4 (Web Text))

Max Activating Token Index: 21

Text #15

Max Range: 1.0007. Min Range: -1.0007

Max Act: 0.8064. Min Act: -0.0000

Data Index: 718673 (C4 (Web Text))

Max Activating Token Index: 143

Text #16

Max Range: 1.0007. Min Range: -1.0007

Max Act: 0.8489. Min Act: -0.0000

Data Index: 987678 (C4 (Web Text))

Max Activating Token Index: 881

Text #17

Max Range: 1.0007. Min Range: -1.0007

Max Act: 0.7768. Min Act: -0.0000

Data Index: 179296 (C4 (Web Text))

Max Activating Token Index: 94

Text #18

Max Range: 1.0007. Min Range: -1.0007

Max Act: 0.7961. Min Act: -0.0000

Data Index: 1137724 (C4 (Web Text))

Max Activating Token Index: 318

Text #19

Max Range: 1.0007. Min Range: -1.0007

Max Act: 0.8402. Min Act: -0.0000

Data Index: 214246 (C4 (Web Text))

Max Activating Token Index: 249