Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1985 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 2.8261. Min Range: -2.8261

Max Act: 2.8261. Min Act: -0.0000

Data Index: 235450 (C4 (Web Text))

Max Activating Token Index: 68

Text #1

Max Range: 2.8261. Min Range: -2.8261

Max Act: 2.5215. Min Act: -0.0000

Data Index: 843523 (C4 (Web Text))

Max Activating Token Index: 405

Text #2

Max Range: 2.8261. Min Range: -2.8261

Max Act: 2.5564. Min Act: -0.0000

Data Index: 1470599 (Python Code)

Max Activating Token Index: 681

Text #3

Max Range: 2.8261. Min Range: -2.8261

Max Act: 2.4753. Min Act: -0.0000

Data Index: 1093132 (C4 (Web Text))

Max Activating Token Index: 900

Text #4

Max Range: 2.8261. Min Range: -2.8261

Max Act: 2.3563. Min Act: -0.0000

Data Index: 1477700 (Python Code)

Max Activating Token Index: 247

Text #5

Max Range: 2.8261. Min Range: -2.8261

Max Act: 2.3618. Min Act: -0.0000

Data Index: 1534272 (Python Code)

Max Activating Token Index: 500

Text #6

Max Range: 2.8261. Min Range: -2.8261

Max Act: 2.2833. Min Act: -0.0000

Data Index: 1431623 (Python Code)

Max Activating Token Index: 356

Text #7

Max Range: 2.8261. Min Range: -2.8261

Max Act: 2.2883. Min Act: -0.0000

Data Index: 192331 (C4 (Web Text))

Max Activating Token Index: 564

Text #8

Max Range: 2.8261. Min Range: -2.8261

Max Act: 2.2551. Min Act: -0.0000

Data Index: 1486717 (Python Code)

Max Activating Token Index: 437

Text #9

Max Range: 2.8261. Min Range: -2.8261

Max Act: 2.1401. Min Act: -0.0000

Data Index: 1074705 (C4 (Web Text))

Max Activating Token Index: 351

Text #10

Max Range: 2.8261. Min Range: -2.8261

Max Act: 2.0974. Min Act: -0.0000

Data Index: 358891 (C4 (Web Text))

Max Activating Token Index: 78

Text #11

Max Range: 2.8261. Min Range: -2.8261

Max Act: 2.0377. Min Act: -0.0000

Data Index: 1272167 (C4 (Web Text))

Max Activating Token Index: 843

Text #12

Max Range: 2.8261. Min Range: -2.8261

Max Act: 2.1207. Min Act: -0.0000

Data Index: 1407669 (Python Code)

Max Activating Token Index: 116

Text #13

Max Range: 2.8261. Min Range: -2.8261

Max Act: 1.8903. Min Act: -0.0000

Data Index: 416697 (C4 (Web Text))

Max Activating Token Index: 739

Text #14

Max Range: 2.8261. Min Range: -2.8261

Max Act: 1.9502. Min Act: -0.0000

Data Index: 496307 (C4 (Web Text))

Max Activating Token Index: 27

Text #15

Max Range: 2.8261. Min Range: -2.8261

Max Act: 1.9318. Min Act: -0.0000

Data Index: 1518631 (Python Code)

Max Activating Token Index: 979

Text #16

Max Range: 2.8261. Min Range: -2.8261

Max Act: 1.7932. Min Act: -0.0000

Data Index: 28870 (C4 (Web Text))

Max Activating Token Index: 844

Text #17

Max Range: 2.8261. Min Range: -2.8261

Max Act: 1.7686. Min Act: -0.0000

Data Index: 805467 (C4 (Web Text))

Max Activating Token Index: 621

Text #18

Max Range: 2.8261. Min Range: -2.8261

Max Act: 1.8192. Min Act: -0.0000

Data Index: 1001867 (C4 (Web Text))

Max Activating Token Index: 721

Text #19

Max Range: 2.8261. Min Range: -2.8261

Max Act: 1.8132. Min Act: -0.0000

Data Index: 17007 (C4 (Web Text))

Max Activating Token Index: 196