Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 236 in Layer 10

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 10.0222. Min Range: -10.0222

Max Act: 10.0222. Min Act: -0.0000

Data Index: 1386665 (Python Code)

Max Activating Token Index: 437

Text #1

Max Range: 10.0222. Min Range: -10.0222

Max Act: 9.4755. Min Act: -0.0000

Data Index: 1600721 (Python Code)

Max Activating Token Index: 729

Text #2

Max Range: 10.0222. Min Range: -10.0222

Max Act: 8.7282. Min Act: -0.0000

Data Index: 1364233 (Python Code)

Max Activating Token Index: 938

Text #3

Max Range: 10.0222. Min Range: -10.0222

Max Act: 7.9221. Min Act: -0.0000

Data Index: 1629403 (Python Code)

Max Activating Token Index: 934

Text #4

Max Range: 10.0222. Min Range: -10.0222

Max Act: 7.3709. Min Act: -0.0000

Data Index: 1521615 (Python Code)

Max Activating Token Index: 880

Text #5

Max Range: 10.0222. Min Range: -10.0222

Max Act: 6.9698. Min Act: -0.0000

Data Index: 1513680 (Python Code)

Max Activating Token Index: 1006

Text #6

Max Range: 10.0222. Min Range: -10.0222

Max Act: 6.9040. Min Act: -0.0000

Data Index: 1586984 (Python Code)

Max Activating Token Index: 958

Text #7

Max Range: 10.0222. Min Range: -10.0222

Max Act: 7.0838. Min Act: -0.0000

Data Index: 1528098 (Python Code)

Max Activating Token Index: 637

Text #8

Max Range: 10.0222. Min Range: -10.0222

Max Act: 7.0421. Min Act: -0.0000

Data Index: 1653386 (Python Code)

Max Activating Token Index: 816

Text #9

Max Range: 10.0222. Min Range: -10.0222

Max Act: 6.8535. Min Act: -0.0000

Data Index: 1401595 (Python Code)

Max Activating Token Index: 879

Text #10

Max Range: 10.0222. Min Range: -10.0222

Max Act: 6.7953. Min Act: -0.0000

Data Index: 1377588 (Python Code)

Max Activating Token Index: 840

Text #11

Max Range: 10.0222. Min Range: -10.0222

Max Act: 6.7123. Min Act: 0.0000

Data Index: 1535859 (Python Code)

Max Activating Token Index: 456

Text #12

Max Range: 10.0222. Min Range: -10.0222

Max Act: 6.6992. Min Act: -0.0000

Data Index: 1519719 (Python Code)

Max Activating Token Index: 508

Text #13

Max Range: 10.0222. Min Range: -10.0222

Max Act: 6.6287. Min Act: -0.0000

Data Index: 1570319 (Python Code)

Max Activating Token Index: 780

Text #14

Max Range: 10.0222. Min Range: -10.0222

Max Act: 6.5976. Min Act: 0.0000

Data Index: 1495482 (Python Code)

Max Activating Token Index: 130

Text #15

Max Range: 10.0222. Min Range: -10.0222

Max Act: 6.3114. Min Act: -0.0000

Data Index: 1555371 (Python Code)

Max Activating Token Index: 801

Text #16

Max Range: 10.0222. Min Range: -10.0222

Max Act: 6.2684. Min Act: -0.0000

Data Index: 1490714 (Python Code)

Max Activating Token Index: 490

Text #17

Max Range: 10.0222. Min Range: -10.0222

Max Act: 6.2770. Min Act: -0.0000

Data Index: 1598896 (Python Code)

Max Activating Token Index: 632

Text #18

Max Range: 10.0222. Min Range: -10.0222

Max Act: 6.2222. Min Act: -0.0000

Data Index: 1509827 (Python Code)

Max Activating Token Index: 769

Text #19

Max Range: 10.0222. Min Range: -10.0222

Max Act: 6.3655. Min Act: -0.0000

Data Index: 1427162 (Python Code)

Max Activating Token Index: 853