Model: SoLU Model: 8 Layers, 4096 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2698 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-8l')

Text #0

Max Range: 0.5136. Min Range: -0.5136

Max Act: 0.4938. Min Act: 0.0001

Data Index: 146565 (C4 (Web Text))

Max Activating Token Index: 684

Text #1

Max Range: 0.5136. Min Range: -0.5136

Max Act: 0.5044. Min Act: 0.0001

Data Index: 1135893 (C4 (Web Text))

Max Activating Token Index: 282

Text #2

Max Range: 0.5136. Min Range: -0.5136

Max Act: 0.5136. Min Act: 0.0001

Data Index: 952201 (C4 (Web Text))

Max Activating Token Index: 647

Text #3

Max Range: 0.5136. Min Range: -0.5136

Max Act: 0.4916. Min Act: 0.0001

Data Index: 1307230 (C4 (Web Text))

Max Activating Token Index: 536

Text #4

Max Range: 0.5136. Min Range: -0.5136

Max Act: 0.4918. Min Act: -0.0000

Data Index: 758507 (C4 (Web Text))

Max Activating Token Index: 315

Text #5

Max Range: 0.5136. Min Range: -0.5136

Max Act: 0.4465. Min Act: -0.0000

Data Index: 523603 (C4 (Web Text))

Max Activating Token Index: 902

Text #6

Max Range: 0.5136. Min Range: -0.5136

Max Act: 0.4758. Min Act: -0.0000

Data Index: 858885 (C4 (Web Text))

Max Activating Token Index: 159

Text #7

Max Range: 0.5136. Min Range: -0.5136

Max Act: 0.4387. Min Act: 0.0001

Data Index: 932919 (C4 (Web Text))

Max Activating Token Index: 556

Text #8

Max Range: 0.5136. Min Range: -0.5136

Max Act: 0.4659. Min Act: -0.0000

Data Index: 1110294 (C4 (Web Text))

Max Activating Token Index: 185

Text #9

Max Range: 0.5136. Min Range: -0.5136

Max Act: 0.4305. Min Act: -0.0000

Data Index: 764332 (C4 (Web Text))

Max Activating Token Index: 837

Text #10

Max Range: 0.5136. Min Range: -0.5136

Max Act: 0.4383. Min Act: 0.0001

Data Index: 1174432 (C4 (Web Text))

Max Activating Token Index: 881

Text #11

Max Range: 0.5136. Min Range: -0.5136

Max Act: 0.4191. Min Act: 0.0001

Data Index: 701328 (C4 (Web Text))

Max Activating Token Index: 181

Text #12

Max Range: 0.5136. Min Range: -0.5136

Max Act: 0.4287. Min Act: -0.0000

Data Index: 1178807 (C4 (Web Text))

Max Activating Token Index: 758

Text #13

Max Range: 0.5136. Min Range: -0.5136

Max Act: 0.4130. Min Act: 0.0001

Data Index: 234895 (C4 (Web Text))

Max Activating Token Index: 248

Text #14

Max Range: 0.5136. Min Range: -0.5136

Max Act: 0.4187. Min Act: 0.0001

Data Index: 942869 (C4 (Web Text))

Max Activating Token Index: 762

Text #15

Max Range: 0.5136. Min Range: -0.5136

Max Act: 0.4262. Min Act: -0.0000

Data Index: 617789 (C4 (Web Text))

Max Activating Token Index: 320

Text #16

Max Range: 0.5136. Min Range: -0.5136

Max Act: 0.4103. Min Act: -0.0000

Data Index: 5091 (C4 (Web Text))

Max Activating Token Index: 952

Text #17

Max Range: 0.5136. Min Range: -0.5136

Max Act: 0.3966. Min Act: -0.0000

Data Index: 375756 (C4 (Web Text))

Max Activating Token Index: 839

Text #18

Max Range: 0.5136. Min Range: -0.5136

Max Act: 0.3900. Min Act: -0.0001

Data Index: 734221 (C4 (Web Text))

Max Activating Token Index: 716

Text #19

Max Range: 0.5136. Min Range: -0.5136

Max Act: 0.3962. Min Act: 0.0001

Data Index: 669360 (C4 (Web Text))

Max Activating Token Index: 960