Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 5622 in Layer 10

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 3.0045. Min Range: -3.0045

Max Act: 3.0045. Min Act: -0.0000

Data Index: 1372194 (Python Code)

Max Activating Token Index: 776

Text #1

Max Range: 3.0045. Min Range: -3.0045

Max Act: 2.8383. Min Act: -0.0000

Data Index: 1562086 (Python Code)

Max Activating Token Index: 620

Text #2

Max Range: 3.0045. Min Range: -3.0045

Max Act: 2.7722. Min Act: -0.0000

Data Index: 1591431 (Python Code)

Max Activating Token Index: 192

Text #3

Max Range: 3.0045. Min Range: -3.0045

Max Act: 2.7394. Min Act: -0.0000

Data Index: 1455152 (Python Code)

Max Activating Token Index: 329

Text #4

Max Range: 3.0045. Min Range: -3.0045

Max Act: 2.6637. Min Act: -0.0000

Data Index: 1420537 (Python Code)

Max Activating Token Index: 440

Text #5

Max Range: 3.0045. Min Range: -3.0045

Max Act: 2.5341. Min Act: -0.0000

Data Index: 1586785 (Python Code)

Max Activating Token Index: 621

Text #6

Max Range: 3.0045. Min Range: -3.0045

Max Act: 2.3052. Min Act: -0.0000

Data Index: 1597921 (Python Code)

Max Activating Token Index: 411

Text #7

Max Range: 3.0045. Min Range: -3.0045

Max Act: 2.3658. Min Act: -0.0000

Data Index: 1418887 (Python Code)

Max Activating Token Index: 755

Text #8

Max Range: 3.0045. Min Range: -3.0045

Max Act: 2.2236. Min Act: -0.0000

Data Index: 1453527 (Python Code)

Max Activating Token Index: 605

Text #9

Max Range: 3.0045. Min Range: -3.0045

Max Act: 1.7346. Min Act: -0.0000

Data Index: 1366082 (Python Code)

Max Activating Token Index: 882

Text #10

Max Range: 3.0045. Min Range: -3.0045

Max Act: 1.6547. Min Act: -0.0000

Data Index: 1579985 (Python Code)

Max Activating Token Index: 66

Text #11

Max Range: 3.0045. Min Range: -3.0045

Max Act: 1.4841. Min Act: -0.0000

Data Index: 1614608 (Python Code)

Max Activating Token Index: 967

Text #12

Max Range: 3.0045. Min Range: -3.0045

Max Act: 1.5073. Min Act: -0.0000

Data Index: 1483803 (Python Code)

Max Activating Token Index: 170

Text #13

Max Range: 3.0045. Min Range: -3.0045

Max Act: 1.5309. Min Act: -0.0000

Data Index: 1450607 (Python Code)

Max Activating Token Index: 228

Text #14

Max Range: 3.0045. Min Range: -3.0045

Max Act: 1.4208. Min Act: -0.0000

Data Index: 1654665 (Python Code)

Max Activating Token Index: 1019

Text #15

Max Range: 3.0045. Min Range: -3.0045

Max Act: 1.5062. Min Act: -0.0000

Data Index: 1402779 (Python Code)

Max Activating Token Index: 72

Text #16

Max Range: 3.0045. Min Range: -3.0045

Max Act: 1.4104. Min Act: -0.0000

Data Index: 1362613 (Python Code)

Max Activating Token Index: 720

Text #17

Max Range: 3.0045. Min Range: -3.0045

Max Act: 1.3772. Min Act: -0.0000

Data Index: 1460457 (Python Code)

Max Activating Token Index: 784

Text #18

Max Range: 3.0045. Min Range: -3.0045

Max Act: 1.3599. Min Act: -0.0000

Data Index: 1631296 (Python Code)

Max Activating Token Index: 250

Text #19

Max Range: 3.0045. Min Range: -3.0045

Max Act: 1.2843. Min Act: -0.0000

Data Index: 1474700 (Python Code)

Max Activating Token Index: 258