Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3044 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 2.1886. Min Range: -2.1886

Max Act: 2.1886. Min Act: -0.0000

Data Index: 1560314 (Python Code)

Max Activating Token Index: 759

Text #1

Max Range: 2.1886. Min Range: -2.1886

Max Act: 2.0175. Min Act: -0.0000

Data Index: 1408783 (Python Code)

Max Activating Token Index: 853

Text #2

Max Range: 2.1886. Min Range: -2.1886

Max Act: 1.8234. Min Act: -0.0000

Data Index: 1612614 (Python Code)

Max Activating Token Index: 186

Text #3

Max Range: 2.1886. Min Range: -2.1886

Max Act: 1.7935. Min Act: -0.0000

Data Index: 1458539 (Python Code)

Max Activating Token Index: 866

Text #4

Max Range: 2.1886. Min Range: -2.1886

Max Act: 1.7764. Min Act: -0.0000

Data Index: 1546310 (Python Code)

Max Activating Token Index: 698

Text #5

Max Range: 2.1886. Min Range: -2.1886

Max Act: 1.7827. Min Act: -0.0000

Data Index: 1569207 (Python Code)

Max Activating Token Index: 304

Text #6

Max Range: 2.1886. Min Range: -2.1886

Max Act: 1.7446. Min Act: -0.0000

Data Index: 1465064 (Python Code)

Max Activating Token Index: 89

Text #7

Max Range: 2.1886. Min Range: -2.1886

Max Act: 1.5434. Min Act: -0.0000

Data Index: 1369666 (Python Code)

Max Activating Token Index: 304

Text #8

Max Range: 2.1886. Min Range: -2.1886

Max Act: 1.7262. Min Act: -0.0000

Data Index: 1653560 (Python Code)

Max Activating Token Index: 138

Text #9

Max Range: 2.1886. Min Range: -2.1886

Max Act: 1.8260. Min Act: -0.0000

Data Index: 1371402 (Python Code)

Max Activating Token Index: 227

Text #10

Max Range: 2.1886. Min Range: -2.1886

Max Act: 1.6123. Min Act: -0.0000

Data Index: 1510180 (Python Code)

Max Activating Token Index: 613

Text #11

Max Range: 2.1886. Min Range: -2.1886

Max Act: 1.7941. Min Act: -0.0000

Data Index: 1509791 (Python Code)

Max Activating Token Index: 929

Text #12

Max Range: 2.1886. Min Range: -2.1886

Max Act: 1.5695. Min Act: -0.0000

Data Index: 1566889 (Python Code)

Max Activating Token Index: 857

Text #13

Max Range: 2.1886. Min Range: -2.1886

Max Act: 1.5422. Min Act: -0.0000

Data Index: 1449564 (Python Code)

Max Activating Token Index: 338

Text #14

Max Range: 2.1886. Min Range: -2.1886

Max Act: 1.5222. Min Act: -0.0000

Data Index: 1577593 (Python Code)

Max Activating Token Index: 114

Text #15

Max Range: 2.1886. Min Range: -2.1886

Max Act: 1.6104. Min Act: -0.0000

Data Index: 1531527 (Python Code)

Max Activating Token Index: 101

Text #16

Max Range: 2.1886. Min Range: -2.1886

Max Act: 1.6017. Min Act: -0.0000

Data Index: 1546994 (Python Code)

Max Activating Token Index: 1012

Text #17

Max Range: 2.1886. Min Range: -2.1886

Max Act: 1.5815. Min Act: -0.0000

Data Index: 1401496 (Python Code)

Max Activating Token Index: 29

Text #18

Max Range: 2.1886. Min Range: -2.1886

Max Act: 1.6669. Min Act: -0.0000

Data Index: 1383093 (Python Code)

Max Activating Token Index: 361

Text #19

Max Range: 2.1886. Min Range: -2.1886

Max Act: 1.6465. Min Act: -0.0000

Data Index: 1491295 (Python Code)

Max Activating Token Index: 789