Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4156 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 0.4944. Min Range: -0.4944

Max Act: 0.4944. Min Act: -0.0000

Data Index: 550351 (C4 (Web Text))

Max Activating Token Index: 363

Text #1

Max Range: 0.4944. Min Range: -0.4944

Max Act: 0.4578. Min Act: -0.0000

Data Index: 618959 (C4 (Web Text))

Max Activating Token Index: 230

Text #2

Max Range: 0.4944. Min Range: -0.4944

Max Act: 0.4372. Min Act: -0.0000

Data Index: 539784 (C4 (Web Text))

Max Activating Token Index: 99

Text #3

Max Range: 0.4944. Min Range: -0.4944

Max Act: 0.4202. Min Act: -0.0000

Data Index: 963172 (C4 (Web Text))

Max Activating Token Index: 404

Text #4

Max Range: 0.4944. Min Range: -0.4944

Max Act: 0.4336. Min Act: -0.0000

Data Index: 541935 (C4 (Web Text))

Max Activating Token Index: 1005

Text #5

Max Range: 0.4944. Min Range: -0.4944

Max Act: 0.4255. Min Act: -0.0000

Data Index: 1259021 (C4 (Web Text))

Max Activating Token Index: 815

Text #6

Max Range: 0.4944. Min Range: -0.4944

Max Act: 0.4042. Min Act: -0.0000

Data Index: 373763 (C4 (Web Text))

Max Activating Token Index: 577

Text #7

Max Range: 0.4944. Min Range: -0.4944

Max Act: 0.3934. Min Act: -0.0000

Data Index: 1303608 (C4 (Web Text))

Max Activating Token Index: 686

Text #8

Max Range: 0.4944. Min Range: -0.4944

Max Act: 0.3935. Min Act: -0.0000

Data Index: 1354400 (C4 (Web Text))

Max Activating Token Index: 887

Text #9

Max Range: 0.4944. Min Range: -0.4944

Max Act: 0.3939. Min Act: -0.0000

Data Index: 1116324 (C4 (Web Text))

Max Activating Token Index: 500

Text #10

Max Range: 0.4944. Min Range: -0.4944

Max Act: 0.4160. Min Act: -0.0000

Data Index: 230730 (C4 (Web Text))

Max Activating Token Index: 955

Text #11

Max Range: 0.4944. Min Range: -0.4944

Max Act: 0.3868. Min Act: -0.0000

Data Index: 296282 (C4 (Web Text))

Max Activating Token Index: 486

Text #12

Max Range: 0.4944. Min Range: -0.4944

Max Act: 0.3666. Min Act: -0.0000

Data Index: 686846 (C4 (Web Text))

Max Activating Token Index: 206

Text #13

Max Range: 0.4944. Min Range: -0.4944

Max Act: 0.3762. Min Act: -0.0000

Data Index: 536377 (C4 (Web Text))

Max Activating Token Index: 944

Text #14

Max Range: 0.4944. Min Range: -0.4944

Max Act: 0.3736. Min Act: -0.0000

Data Index: 183259 (C4 (Web Text))

Max Activating Token Index: 120

Text #15

Max Range: 0.4944. Min Range: -0.4944

Max Act: 0.3939. Min Act: -0.0000

Data Index: 1151884 (C4 (Web Text))

Max Activating Token Index: 941

Text #16

Max Range: 0.4944. Min Range: -0.4944

Max Act: 0.3703. Min Act: -0.0000

Data Index: 229302 (C4 (Web Text))

Max Activating Token Index: 282

Text #17

Max Range: 0.4944. Min Range: -0.4944

Max Act: 0.3782. Min Act: -0.0000

Data Index: 768957 (C4 (Web Text))

Max Activating Token Index: 220

Text #18

Max Range: 0.4944. Min Range: -0.4944

Max Act: 0.3491. Min Act: -0.0000

Data Index: 1163020 (C4 (Web Text))

Max Activating Token Index: 878

Text #19

Max Range: 0.4944. Min Range: -0.4944

Max Act: 0.3593. Min Act: -0.0000

Data Index: 283399 (C4 (Web Text))

Max Activating Token Index: 61