Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 2011 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 1.2166. Min Range: -1.2166

Max Act: 1.2166. Min Act: -0.0000

Data Index: 1684426 (The Pile)

Max Activating Token Index: 122

Text #1

Max Range: 1.2166. Min Range: -1.2166

Max Act: 0.9884. Min Act: -0.0000

Data Index: 117658 (The Pile)

Max Activating Token Index: 250

Text #2

Max Range: 1.2166. Min Range: -1.2166

Max Act: 0.8931. Min Act: -0.0000

Data Index: 1617749 (The Pile)

Max Activating Token Index: 505

Text #3

Max Range: 1.2166. Min Range: -1.2166

Max Act: 0.8999. Min Act: -0.0000

Data Index: 1999921 (The Pile)

Max Activating Token Index: 135

Text #4

Max Range: 1.2166. Min Range: -1.2166

Max Act: 0.8543. Min Act: -0.0000

Data Index: 36140 (The Pile)

Max Activating Token Index: 192

Text #5

Max Range: 1.2166. Min Range: -1.2166

Max Act: 0.9551. Min Act: -0.0000

Data Index: 1375882 (The Pile)

Max Activating Token Index: 151

Text #6

Max Range: 1.2166. Min Range: -1.2166

Max Act: 0.8646. Min Act: -0.0000

Data Index: 1100866 (The Pile)

Max Activating Token Index: 287

Text #7

Max Range: 1.2166. Min Range: -1.2166

Max Act: 0.8468. Min Act: -0.0000

Data Index: 394700 (The Pile)

Max Activating Token Index: 214

Text #8

Max Range: 1.2166. Min Range: -1.2166

Max Act: 0.8381. Min Act: -0.0000

Data Index: 1513459 (The Pile)

Max Activating Token Index: 161

Text #9

Max Range: 1.2166. Min Range: -1.2166

Max Act: 0.8570. Min Act: -0.0000

Data Index: 1142947 (The Pile)

Max Activating Token Index: 189

Text #10

Max Range: 1.2166. Min Range: -1.2166

Max Act: 0.8320. Min Act: -0.0000

Data Index: 688180 (The Pile)

Max Activating Token Index: 214

Text #11

Max Range: 1.2166. Min Range: -1.2166

Max Act: 0.8201. Min Act: -0.0000

Data Index: 907039 (The Pile)

Max Activating Token Index: 247

Text #12

Max Range: 1.2166. Min Range: -1.2166

Max Act: 0.8153. Min Act: -0.0000

Data Index: 549131 (The Pile)

Max Activating Token Index: 188

Text #13

Max Range: 1.2166. Min Range: -1.2166

Max Act: 0.8123. Min Act: -0.0000

Data Index: 439353 (The Pile)

Max Activating Token Index: 140

Text #14

Max Range: 1.2166. Min Range: -1.2166

Max Act: 0.7709. Min Act: -0.0000

Data Index: 48481 (The Pile)

Max Activating Token Index: 146

Text #15

Max Range: 1.2166. Min Range: -1.2166

Max Act: 0.7862. Min Act: -0.0000

Data Index: 490784 (The Pile)

Max Activating Token Index: 186

Text #16

Max Range: 1.2166. Min Range: -1.2166

Max Act: 0.7304. Min Act: -0.0000

Data Index: 1507132 (The Pile)

Max Activating Token Index: 597

Text #17

Max Range: 1.2166. Min Range: -1.2166

Max Act: 0.7461. Min Act: -0.0000

Data Index: 1692347 (The Pile)

Max Activating Token Index: 449

Text #18

Max Range: 1.2166. Min Range: -1.2166

Max Act: 0.7437. Min Act: -0.0000

Data Index: 946277 (The Pile)

Max Activating Token Index: 396

Text #19

Max Range: 1.2166. Min Range: -1.2166

Max Act: 0.6928. Min Act: -0.0000

Data Index: 462615 (The Pile)

Max Activating Token Index: 251