Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3025 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.2793. Min Range: -0.2793

Max Act: 0.2793. Min Act: -0.0000

Data Index: 658013 (C4 (Web Text))

Max Activating Token Index: 60

Text #1

Max Range: 0.2793. Min Range: -0.2793

Max Act: 0.2266. Min Act: -0.0000

Data Index: 745651 (C4 (Web Text))

Max Activating Token Index: 492

Text #2

Max Range: 0.2793. Min Range: -0.2793

Max Act: 0.2233. Min Act: -0.0000

Data Index: 277115 (C4 (Web Text))

Max Activating Token Index: 560

Text #3

Max Range: 0.2793. Min Range: -0.2793

Max Act: 0.2108. Min Act: -0.0000

Data Index: 468974 (C4 (Web Text))

Max Activating Token Index: 698

Text #4

Max Range: 0.2793. Min Range: -0.2793

Max Act: 0.2214. Min Act: -0.0000

Data Index: 109271 (C4 (Web Text))

Max Activating Token Index: 356

Text #5

Max Range: 0.2793. Min Range: -0.2793

Max Act: 0.2138. Min Act: -0.0000

Data Index: 443788 (C4 (Web Text))

Max Activating Token Index: 363

Text #6

Max Range: 0.2793. Min Range: -0.2793

Max Act: 0.2187. Min Act: -0.0000

Data Index: 707104 (C4 (Web Text))

Max Activating Token Index: 541

Text #7

Max Range: 0.2793. Min Range: -0.2793

Max Act: 0.2084. Min Act: -0.0000

Data Index: 933122 (C4 (Web Text))

Max Activating Token Index: 343

Text #8

Max Range: 0.2793. Min Range: -0.2793

Max Act: 0.2051. Min Act: -0.0000

Data Index: 16857 (C4 (Web Text))

Max Activating Token Index: 104

Text #9

Max Range: 0.2793. Min Range: -0.2793

Max Act: 0.2017. Min Act: -0.0000

Data Index: 588173 (C4 (Web Text))

Max Activating Token Index: 30

Text #10

Max Range: 0.2793. Min Range: -0.2793

Max Act: 0.2054. Min Act: -0.0000

Data Index: 480032 (C4 (Web Text))

Max Activating Token Index: 793

Text #11

Max Range: 0.2793. Min Range: -0.2793

Max Act: 0.2125. Min Act: -0.0000

Data Index: 1034551 (C4 (Web Text))

Max Activating Token Index: 780

Text #12

Max Range: 0.2793. Min Range: -0.2793

Max Act: 0.1960. Min Act: -0.0000

Data Index: 233713 (C4 (Web Text))

Max Activating Token Index: 637

Text #13

Max Range: 0.2793. Min Range: -0.2793

Max Act: 0.2008. Min Act: -0.0000

Data Index: 1098785 (C4 (Web Text))

Max Activating Token Index: 584

Text #14

Max Range: 0.2793. Min Range: -0.2793

Max Act: 0.1947. Min Act: -0.0000

Data Index: 1283632 (C4 (Web Text))

Max Activating Token Index: 857

Text #15

Max Range: 0.2793. Min Range: -0.2793

Max Act: 0.1987. Min Act: -0.0000

Data Index: 1174404 (C4 (Web Text))

Max Activating Token Index: 188

Text #16

Max Range: 0.2793. Min Range: -0.2793

Max Act: 0.1910. Min Act: -0.0000

Data Index: 968522 (C4 (Web Text))

Max Activating Token Index: 92

Text #17

Max Range: 0.2793. Min Range: -0.2793

Max Act: 0.1844. Min Act: -0.0000

Data Index: 39429 (C4 (Web Text))

Max Activating Token Index: 83

Text #18

Max Range: 0.2793. Min Range: -0.2793

Max Act: 0.2020. Min Act: -0.0000

Data Index: 1324221 (C4 (Web Text))

Max Activating Token Index: 167

Text #19

Max Range: 0.2793. Min Range: -0.2793

Max Act: 0.1882. Min Act: -0.0000

Data Index: 485778 (C4 (Web Text))

Max Activating Token Index: 821