Model: SoLU Model: 8 Layers, 4096 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 51 in Layer 4

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-8l')

Text #0

Max Range: 0.1362. Min Range: -0.1362

Max Act: 0.1362. Min Act: -0.0001

Data Index: 803828 (C4 (Web Text))

Max Activating Token Index: 351

Text #1

Max Range: 0.1362. Min Range: -0.1362

Max Act: 0.1347. Min Act: -0.0001

Data Index: 753345 (C4 (Web Text))

Max Activating Token Index: 883

Text #2

Max Range: 0.1362. Min Range: -0.1362

Max Act: 0.1233. Min Act: -0.0001

Data Index: 1245388 (C4 (Web Text))

Max Activating Token Index: 522

Text #3

Max Range: 0.1362. Min Range: -0.1362

Max Act: 0.1113. Min Act: -0.0001

Data Index: 1520000 (Python Code)

Max Activating Token Index: 303

Text #4

Max Range: 0.1362. Min Range: -0.1362

Max Act: 0.1034. Min Act: -0.0001

Data Index: 599165 (C4 (Web Text))

Max Activating Token Index: 941

Text #5

Max Range: 0.1362. Min Range: -0.1362

Max Act: 0.0991. Min Act: -0.0001

Data Index: 1234168 (C4 (Web Text))

Max Activating Token Index: 50

Text #6

Max Range: 0.1362. Min Range: -0.1362

Max Act: 0.0982. Min Act: -0.0001

Data Index: 1442965 (Python Code)

Max Activating Token Index: 353

Text #7

Max Range: 0.1362. Min Range: -0.1362

Max Act: 0.0911. Min Act: -0.0001

Data Index: 1534315 (Python Code)

Max Activating Token Index: 836

Text #8

Max Range: 0.1362. Min Range: -0.1362

Max Act: 0.1124. Min Act: -0.0001

Data Index: 1439619 (Python Code)

Max Activating Token Index: 606

Text #9

Max Range: 0.1362. Min Range: -0.1362

Max Act: 0.1006. Min Act: -0.0001

Data Index: 1425160 (Python Code)

Max Activating Token Index: 968

Text #10

Max Range: 0.1362. Min Range: -0.1362

Max Act: 0.0983. Min Act: -0.0001

Data Index: 1626337 (Python Code)

Max Activating Token Index: 804

Text #11

Max Range: 0.1362. Min Range: -0.1362

Max Act: 0.0902. Min Act: -0.0001

Data Index: 1396828 (Python Code)

Max Activating Token Index: 646

Text #12

Max Range: 0.1362. Min Range: -0.1362

Max Act: 0.0889. Min Act: -0.0001

Data Index: 1628593 (Python Code)

Max Activating Token Index: 713

Text #13

Max Range: 0.1362. Min Range: -0.1362

Max Act: 0.0987. Min Act: -0.0001

Data Index: 1588060 (Python Code)

Max Activating Token Index: 803

Text #14

Max Range: 0.1362. Min Range: -0.1362

Max Act: 0.0895. Min Act: -0.0001

Data Index: 1593103 (Python Code)

Max Activating Token Index: 549

Text #15

Max Range: 0.1362. Min Range: -0.1362

Max Act: 0.0976. Min Act: -0.0001

Data Index: 1557169 (Python Code)

Max Activating Token Index: 896

Text #16

Max Range: 0.1362. Min Range: -0.1362

Max Act: 0.0949. Min Act: -0.0001

Data Index: 1438784 (Python Code)

Max Activating Token Index: 359

Text #17

Max Range: 0.1362. Min Range: -0.1362

Max Act: 0.0977. Min Act: -0.0001

Data Index: 1522228 (Python Code)

Max Activating Token Index: 422

Text #18

Max Range: 0.1362. Min Range: -0.1362

Max Act: 0.0975. Min Act: -0.0001

Data Index: 1525236 (Python Code)

Max Activating Token Index: 560

Text #19

Max Range: 0.1362. Min Range: -0.1362

Max Act: 0.0909. Min Act: -0.0001

Data Index: 1638583 (Python Code)

Max Activating Token Index: 714