Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2007 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 4.4101. Min Range: -4.4101

Max Act: 4.4101. Min Act: -0.0000

Data Index: 1624432 (Python Code)

Max Activating Token Index: 352

Text #1

Max Range: 4.4101. Min Range: -4.4101

Max Act: 4.0621. Min Act: -0.0000

Data Index: 1512633 (Python Code)

Max Activating Token Index: 223

Text #2

Max Range: 4.4101. Min Range: -4.4101

Max Act: 4.0115. Min Act: -0.0000

Data Index: 1466440 (Python Code)

Max Activating Token Index: 862

Text #3

Max Range: 4.4101. Min Range: -4.4101

Max Act: 3.9959. Min Act: -0.0000

Data Index: 1415598 (Python Code)

Max Activating Token Index: 1001

Text #4

Max Range: 4.4101. Min Range: -4.4101

Max Act: 3.7866. Min Act: -0.0000

Data Index: 1568683 (Python Code)

Max Activating Token Index: 356

Text #5

Max Range: 4.4101. Min Range: -4.4101

Max Act: 3.7678. Min Act: -0.0000

Data Index: 1577020 (Python Code)

Max Activating Token Index: 992

Text #6

Max Range: 4.4101. Min Range: -4.4101

Max Act: 3.9115. Min Act: -0.0000

Data Index: 1572406 (Python Code)

Max Activating Token Index: 882

Text #7

Max Range: 4.4101. Min Range: -4.4101

Max Act: 4.0446. Min Act: -0.0000

Data Index: 1485744 (Python Code)

Max Activating Token Index: 913

Text #8

Max Range: 4.4101. Min Range: -4.4101

Max Act: 3.6329. Min Act: -0.0000

Data Index: 1516588 (Python Code)

Max Activating Token Index: 573

Text #9

Max Range: 4.4101. Min Range: -4.4101

Max Act: 3.6359. Min Act: -0.0000

Data Index: 1410493 (Python Code)

Max Activating Token Index: 796

Text #10

Max Range: 4.4101. Min Range: -4.4101

Max Act: 3.5904. Min Act: -0.0000

Data Index: 1545834 (Python Code)

Max Activating Token Index: 831

Text #11

Max Range: 4.4101. Min Range: -4.4101

Max Act: 3.5871. Min Act: -0.0000

Data Index: 1653214 (Python Code)

Max Activating Token Index: 855

Text #12

Max Range: 4.4101. Min Range: -4.4101

Max Act: 3.4071. Min Act: -0.0000

Data Index: 1406652 (Python Code)

Max Activating Token Index: 127

Text #13

Max Range: 4.4101. Min Range: -4.4101

Max Act: 3.5428. Min Act: -0.0000

Data Index: 1386608 (Python Code)

Max Activating Token Index: 697

Text #14

Max Range: 4.4101. Min Range: -4.4101

Max Act: 3.4950. Min Act: -0.0000

Data Index: 1480639 (Python Code)

Max Activating Token Index: 535

Text #15

Max Range: 4.4101. Min Range: -4.4101

Max Act: 3.6709. Min Act: -0.0000

Data Index: 1437035 (Python Code)

Max Activating Token Index: 946

Text #16

Max Range: 4.4101. Min Range: -4.4101

Max Act: 3.4576. Min Act: -0.0000

Data Index: 1517745 (Python Code)

Max Activating Token Index: 754

Text #17

Max Range: 4.4101. Min Range: -4.4101

Max Act: 3.2932. Min Act: -0.0000

Data Index: 1443280 (Python Code)

Max Activating Token Index: 240

Text #18

Max Range: 4.4101. Min Range: -4.4101

Max Act: 3.4511. Min Act: -0.0000

Data Index: 1428439 (Python Code)

Max Activating Token Index: 37

Text #19

Max Range: 4.4101. Min Range: -4.4101

Max Act: 3.4324. Min Act: -0.0000

Data Index: 1457076 (Python Code)

Max Activating Token Index: 850