Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4432 in Layer 11

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 2.9881. Min Range: -2.9881

Max Act: 2.9222. Min Act: -0.0000

Data Index: 1600631 (Python Code)

Max Activating Token Index: 563

Text #1

Max Range: 2.9881. Min Range: -2.9881

Max Act: 2.7886. Min Act: -0.0000

Data Index: 1650021 (Python Code)

Max Activating Token Index: 957

Text #2

Max Range: 2.9881. Min Range: -2.9881

Max Act: 2.6075. Min Act: -0.0000

Data Index: 1427753 (Python Code)

Max Activating Token Index: 815

Text #3

Max Range: 2.9881. Min Range: -2.9881

Max Act: 2.9881. Min Act: -0.0000

Data Index: 1422507 (Python Code)

Max Activating Token Index: 236

Text #4

Max Range: 2.9881. Min Range: -2.9881

Max Act: 2.6396. Min Act: -0.0000

Data Index: 1529260 (Python Code)

Max Activating Token Index: 639

Text #5

Max Range: 2.9881. Min Range: -2.9881

Max Act: 2.6017. Min Act: -0.0000

Data Index: 1523949 (Python Code)

Max Activating Token Index: 937

Text #6

Max Range: 2.9881. Min Range: -2.9881

Max Act: 2.5024. Min Act: -0.0000

Data Index: 1503088 (Python Code)

Max Activating Token Index: 509

Text #7

Max Range: 2.9881. Min Range: -2.9881

Max Act: 2.3493. Min Act: -0.0000

Data Index: 1373425 (Python Code)

Max Activating Token Index: 334

Text #8

Max Range: 2.9881. Min Range: -2.9881

Max Act: 2.4709. Min Act: -0.0000

Data Index: 1545836 (Python Code)

Max Activating Token Index: 157

Text #9

Max Range: 2.9881. Min Range: -2.9881

Max Act: 2.3435. Min Act: -0.0000

Data Index: 1488744 (Python Code)

Max Activating Token Index: 79

Text #10

Max Range: 2.9881. Min Range: -2.9881

Max Act: 2.1203. Min Act: -0.0000

Data Index: 1596736 (Python Code)

Max Activating Token Index: 101

Text #11

Max Range: 2.9881. Min Range: -2.9881

Max Act: 2.4178. Min Act: -0.0000

Data Index: 1370359 (Python Code)

Max Activating Token Index: 153

Text #12

Max Range: 2.9881. Min Range: -2.9881

Max Act: 2.3662. Min Act: -0.0000

Data Index: 1643036 (Python Code)

Max Activating Token Index: 208

Text #13

Max Range: 2.9881. Min Range: -2.9881

Max Act: 2.3741. Min Act: -0.0000

Data Index: 1488867 (Python Code)

Max Activating Token Index: 848

Text #14

Max Range: 2.9881. Min Range: -2.9881

Max Act: 1.9848. Min Act: -0.0000

Data Index: 1521335 (Python Code)

Max Activating Token Index: 741

Text #15

Max Range: 2.9881. Min Range: -2.9881

Max Act: 2.1470. Min Act: -0.0000

Data Index: 1437986 (Python Code)

Max Activating Token Index: 752

Text #16

Max Range: 2.9881. Min Range: -2.9881

Max Act: 2.2849. Min Act: -0.0000

Data Index: 1595020 (Python Code)

Max Activating Token Index: 291

Text #17

Max Range: 2.9881. Min Range: -2.9881

Max Act: 2.0404. Min Act: -0.0000

Data Index: 1436874 (Python Code)

Max Activating Token Index: 861

Text #18

Max Range: 2.9881. Min Range: -2.9881

Max Act: 2.0858. Min Act: -0.0000

Data Index: 1427299 (Python Code)

Max Activating Token Index: 430

Text #19

Max Range: 2.9881. Min Range: -2.9881

Max Act: 2.0715. Min Act: -0.0000

Data Index: 1364405 (Python Code)

Max Activating Token Index: 495