Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4023 in Layer 4

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.1701. Min Range: -1.1701

Max Act: 1.1701. Min Act: -0.0000

Data Index: 1603191 (Python Code)

Max Activating Token Index: 85

Text #1

Max Range: 1.1701. Min Range: -1.1701

Max Act: 1.0699. Min Act: -0.0000

Data Index: 1585939 (Python Code)

Max Activating Token Index: 50

Text #2

Max Range: 1.1701. Min Range: -1.1701

Max Act: 1.0810. Min Act: -0.0000

Data Index: 1459047 (Python Code)

Max Activating Token Index: 970

Text #3

Max Range: 1.1701. Min Range: -1.1701

Max Act: 0.9403. Min Act: -0.0000

Data Index: 1574541 (Python Code)

Max Activating Token Index: 424

Text #4

Max Range: 1.1701. Min Range: -1.1701

Max Act: 0.9445. Min Act: -0.0000

Data Index: 1589171 (Python Code)

Max Activating Token Index: 384

Text #5

Max Range: 1.1701. Min Range: -1.1701

Max Act: 0.9979. Min Act: -0.0000

Data Index: 1387775 (Python Code)

Max Activating Token Index: 710

Text #6

Max Range: 1.1701. Min Range: -1.1701

Max Act: 0.9253. Min Act: -0.0000

Data Index: 1462239 (Python Code)

Max Activating Token Index: 263

Text #7

Max Range: 1.1701. Min Range: -1.1701

Max Act: 0.7748. Min Act: -0.0000

Data Index: 1654386 (Python Code)

Max Activating Token Index: 278

Text #8

Max Range: 1.1701. Min Range: -1.1701

Max Act: 0.8342. Min Act: -0.0000

Data Index: 1457365 (Python Code)

Max Activating Token Index: 569

Text #9

Max Range: 1.1701. Min Range: -1.1701

Max Act: 0.7920. Min Act: -0.0000

Data Index: 1592207 (Python Code)

Max Activating Token Index: 168

Text #10

Max Range: 1.1701. Min Range: -1.1701

Max Act: 0.7703. Min Act: -0.0000

Data Index: 1542652 (Python Code)

Max Activating Token Index: 165

Text #11

Max Range: 1.1701. Min Range: -1.1701

Max Act: 0.7226. Min Act: -0.0000

Data Index: 1652747 (Python Code)

Max Activating Token Index: 218

Text #12

Max Range: 1.1701. Min Range: -1.1701

Max Act: 0.8408. Min Act: -0.0000

Data Index: 1608884 (Python Code)

Max Activating Token Index: 770

Text #13

Max Range: 1.1701. Min Range: -1.1701

Max Act: 0.8058. Min Act: -0.0000

Data Index: 1655796 (Python Code)

Max Activating Token Index: 1002

Text #14

Max Range: 1.1701. Min Range: -1.1701

Max Act: 0.7416. Min Act: -0.0000

Data Index: 1375516 (Python Code)

Max Activating Token Index: 492

Text #15

Max Range: 1.1701. Min Range: -1.1701

Max Act: 0.8013. Min Act: -0.0000

Data Index: 1360435 (Python Code)

Max Activating Token Index: 682

Text #16

Max Range: 1.1701. Min Range: -1.1701

Max Act: 0.7181. Min Act: -0.0000

Data Index: 1423888 (Python Code)

Max Activating Token Index: 200

Text #17

Max Range: 1.1701. Min Range: -1.1701

Max Act: 0.8299. Min Act: -0.0000

Data Index: 1464954 (Python Code)

Max Activating Token Index: 753

Text #18

Max Range: 1.1701. Min Range: -1.1701

Max Act: 0.7496. Min Act: -0.0000

Data Index: 1588072 (Python Code)

Max Activating Token Index: 169

Text #19

Max Range: 1.1701. Min Range: -1.1701

Max Act: 0.7533. Min Act: -0.0000

Data Index: 1515848 (Python Code)

Max Activating Token Index: 61