Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1985 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.4784. Min Range: -1.4784

Max Act: 1.4784. Min Act: -0.0000

Data Index: 1506919 (Python Code)

Max Activating Token Index: 635

Text #1

Max Range: 1.4784. Min Range: -1.4784

Max Act: 1.3262. Min Act: -0.0000

Data Index: 1545315 (Python Code)

Max Activating Token Index: 709

Text #2

Max Range: 1.4784. Min Range: -1.4784

Max Act: 1.2975. Min Act: -0.0000

Data Index: 1506817 (Python Code)

Max Activating Token Index: 805

Text #3

Max Range: 1.4784. Min Range: -1.4784

Max Act: 1.2406. Min Act: -0.0000

Data Index: 575075 (C4 (Web Text))

Max Activating Token Index: 329

Text #4

Max Range: 1.4784. Min Range: -1.4784

Max Act: 1.2444. Min Act: -0.0000

Data Index: 1609677 (Python Code)

Max Activating Token Index: 584

Text #5

Max Range: 1.4784. Min Range: -1.4784

Max Act: 1.2293. Min Act: -0.0000

Data Index: 1607463 (Python Code)

Max Activating Token Index: 554

Text #6

Max Range: 1.4784. Min Range: -1.4784

Max Act: 1.1678. Min Act: -0.0000

Data Index: 1631306 (Python Code)

Max Activating Token Index: 672

Text #7

Max Range: 1.4784. Min Range: -1.4784

Max Act: 1.1583. Min Act: -0.0000

Data Index: 990527 (C4 (Web Text))

Max Activating Token Index: 776

Text #8

Max Range: 1.4784. Min Range: -1.4784

Max Act: 1.1812. Min Act: -0.0000

Data Index: 1073059 (C4 (Web Text))

Max Activating Token Index: 272

Text #9

Max Range: 1.4784. Min Range: -1.4784

Max Act: 1.1669. Min Act: -0.0000

Data Index: 1453226 (Python Code)

Max Activating Token Index: 531

Text #10

Max Range: 1.4784. Min Range: -1.4784

Max Act: 1.1618. Min Act: -0.0000

Data Index: 1131360 (C4 (Web Text))

Max Activating Token Index: 956

Text #11

Max Range: 1.4784. Min Range: -1.4784

Max Act: 1.2242. Min Act: -0.0000

Data Index: 1595767 (Python Code)

Max Activating Token Index: 766

Text #12

Max Range: 1.4784. Min Range: -1.4784

Max Act: 1.1129. Min Act: -0.0000

Data Index: 1599225 (Python Code)

Max Activating Token Index: 555

Text #13

Max Range: 1.4784. Min Range: -1.4784

Max Act: 1.1000. Min Act: -0.0000

Data Index: 1645597 (Python Code)

Max Activating Token Index: 144

Text #14

Max Range: 1.4784. Min Range: -1.4784

Max Act: 1.0573. Min Act: -0.0000

Data Index: 1617555 (Python Code)

Max Activating Token Index: 638

Text #15

Max Range: 1.4784. Min Range: -1.4784

Max Act: 1.1238. Min Act: -0.0000

Data Index: 302288 (C4 (Web Text))

Max Activating Token Index: 21

Text #16

Max Range: 1.4784. Min Range: -1.4784

Max Act: 1.0485. Min Act: -0.0000

Data Index: 1549597 (Python Code)

Max Activating Token Index: 776

Text #17

Max Range: 1.4784. Min Range: -1.4784

Max Act: 1.1045. Min Act: -0.0000

Data Index: 1522071 (Python Code)

Max Activating Token Index: 705

Text #18

Max Range: 1.4784. Min Range: -1.4784

Max Act: 1.1374. Min Act: -0.0000

Data Index: 115134 (C4 (Web Text))

Max Activating Token Index: 1007

Text #19

Max Range: 1.4784. Min Range: -1.4784

Max Act: 1.1076. Min Act: -0.0000

Data Index: 393595 (C4 (Web Text))

Max Activating Token Index: 361