Model: GELU Model: 3 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 402 in Layer 1

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('gelu-3l')

Text #0

Max Range: 3.7186. Min Range: -3.7186

Max Act: 3.7186. Min Act: -0.1700

Data Index: 536053 (C4 (Web Text))

Max Activating Token Index: 595

Text #1

Max Range: 3.7186. Min Range: -3.7186

Max Act: 3.6247. Min Act: -0.1700

Data Index: 305729 (C4 (Web Text))

Max Activating Token Index: 658

Text #2

Max Range: 3.7186. Min Range: -3.7186

Max Act: 3.5307. Min Act: -0.1700

Data Index: 889585 (C4 (Web Text))

Max Activating Token Index: 646

Text #3

Max Range: 3.7186. Min Range: -3.7186

Max Act: 3.5620. Min Act: -0.1700

Data Index: 929250 (C4 (Web Text))

Max Activating Token Index: 173

Text #4

Max Range: 3.7186. Min Range: -3.7186

Max Act: 3.5620. Min Act: -0.1700

Data Index: 150035 (C4 (Web Text))

Max Activating Token Index: 501

Text #5

Max Range: 3.7186. Min Range: -3.7186

Max Act: 3.4994. Min Act: -0.1700

Data Index: 958768 (C4 (Web Text))

Max Activating Token Index: 70

Text #6

Max Range: 3.7186. Min Range: -3.7186

Max Act: 3.4367. Min Act: -0.1700

Data Index: 1091866 (C4 (Web Text))

Max Activating Token Index: 758

Text #7

Max Range: 3.7186. Min Range: -3.7186

Max Act: 3.4994. Min Act: -0.1700

Data Index: 1165974 (C4 (Web Text))

Max Activating Token Index: 642

Text #8

Max Range: 3.7186. Min Range: -3.7186

Max Act: 3.4994. Min Act: -0.1700

Data Index: 511597 (C4 (Web Text))

Max Activating Token Index: 770

Text #9

Max Range: 3.7186. Min Range: -3.7186

Max Act: 3.4994. Min Act: -0.1700

Data Index: 1301886 (C4 (Web Text))

Max Activating Token Index: 902

Text #10

Max Range: 3.7186. Min Range: -3.7186

Max Act: 3.4680. Min Act: -0.1700

Data Index: 152726 (C4 (Web Text))

Max Activating Token Index: 241

Text #11

Max Range: 3.7186. Min Range: -3.7186

Max Act: 3.4837. Min Act: -0.1700

Data Index: 364936 (C4 (Web Text))

Max Activating Token Index: 503

Text #12

Max Range: 3.7186. Min Range: -3.7186

Max Act: 3.4210. Min Act: -0.1700

Data Index: 169793 (C4 (Web Text))

Max Activating Token Index: 910

Text #13

Max Range: 3.7186. Min Range: -3.7186

Max Act: 3.4367. Min Act: -0.1700

Data Index: 1139972 (C4 (Web Text))

Max Activating Token Index: 971

Text #14

Max Range: 3.7186. Min Range: -3.7186

Max Act: 3.3426. Min Act: -0.1700

Data Index: 1082944 (C4 (Web Text))

Max Activating Token Index: 990

Text #15

Max Range: 3.7186. Min Range: -3.7186

Max Act: 3.3739. Min Act: -0.1700

Data Index: 931956 (C4 (Web Text))

Max Activating Token Index: 774

Text #16

Max Range: 3.7186. Min Range: -3.7186

Max Act: 3.3739. Min Act: -0.1700

Data Index: 916037 (C4 (Web Text))

Max Activating Token Index: 160

Text #17

Max Range: 3.7186. Min Range: -3.7186

Max Act: 3.3739. Min Act: -0.1700

Data Index: 330288 (C4 (Web Text))

Max Activating Token Index: 474

Text #18

Max Range: 3.7186. Min Range: -3.7186

Max Act: 3.3583. Min Act: -0.1700

Data Index: 660804 (C4 (Web Text))

Max Activating Token Index: 891

Text #19

Max Range: 3.7186. Min Range: -3.7186

Max Act: 3.3739. Min Act: -0.1700

Data Index: 693471 (C4 (Web Text))

Max Activating Token Index: 839