Model: GELU Model: 4 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1280 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('gelu-4l')

Text #0

Max Range: 3.4870. Min Range: -3.4870

Max Act: 3.4870. Min Act: -0.1700

Data Index: 514638 (C4 (Web Text))

Max Activating Token Index: 753

Text #1

Max Range: 3.4870. Min Range: -3.4870

Max Act: 3.4087. Min Act: -0.1700

Data Index: 811805 (C4 (Web Text))

Max Activating Token Index: 918

Text #2

Max Range: 3.4870. Min Range: -3.4870

Max Act: 3.3773. Min Act: -0.1700

Data Index: 261967 (C4 (Web Text))

Max Activating Token Index: 404

Text #3

Max Range: 3.4870. Min Range: -3.4870

Max Act: 3.4087. Min Act: -0.1700

Data Index: 1230601 (C4 (Web Text))

Max Activating Token Index: 588

Text #4

Max Range: 3.4870. Min Range: -3.4870

Max Act: 3.3773. Min Act: -0.1700

Data Index: 891743 (C4 (Web Text))

Max Activating Token Index: 995

Text #5

Max Range: 3.4870. Min Range: -3.4870

Max Act: 3.3773. Min Act: -0.1700

Data Index: 758373 (C4 (Web Text))

Max Activating Token Index: 569

Text #6

Max Range: 3.4870. Min Range: -3.4870

Max Act: 3.4087. Min Act: -0.1700

Data Index: 328807 (C4 (Web Text))

Max Activating Token Index: 182

Text #7

Max Range: 3.4870. Min Range: -3.4870

Max Act: 3.3930. Min Act: -0.1700

Data Index: 707470 (C4 (Web Text))

Max Activating Token Index: 351

Text #8

Max Range: 3.4870. Min Range: -3.4870

Max Act: 3.3773. Min Act: -0.1700

Data Index: 476350 (C4 (Web Text))

Max Activating Token Index: 264

Text #9

Max Range: 3.4870. Min Range: -3.4870

Max Act: 3.3930. Min Act: -0.1700

Data Index: 277687 (C4 (Web Text))

Max Activating Token Index: 545

Text #10

Max Range: 3.4870. Min Range: -3.4870

Max Act: 3.3459. Min Act: -0.1700

Data Index: 859049 (C4 (Web Text))

Max Activating Token Index: 601

Text #11

Max Range: 3.4870. Min Range: -3.4870

Max Act: 3.3616. Min Act: -0.1700

Data Index: 554510 (C4 (Web Text))

Max Activating Token Index: 405

Text #12

Max Range: 3.4870. Min Range: -3.4870

Max Act: 3.2988. Min Act: -0.1700

Data Index: 983013 (C4 (Web Text))

Max Activating Token Index: 691

Text #13

Max Range: 3.4870. Min Range: -3.4870

Max Act: 3.3459. Min Act: -0.1700

Data Index: 988763 (C4 (Web Text))

Max Activating Token Index: 366

Text #14

Max Range: 3.4870. Min Range: -3.4870

Max Act: 3.3145. Min Act: -0.1700

Data Index: 386251 (C4 (Web Text))

Max Activating Token Index: 264

Text #15

Max Range: 3.4870. Min Range: -3.4870

Max Act: 3.2988. Min Act: -0.1700

Data Index: 966852 (C4 (Web Text))

Max Activating Token Index: 957

Text #16

Max Range: 3.4870. Min Range: -3.4870

Max Act: 3.2831. Min Act: -0.1700

Data Index: 924235 (C4 (Web Text))

Max Activating Token Index: 462

Text #17

Max Range: 3.4870. Min Range: -3.4870

Max Act: 3.3145. Min Act: -0.1700

Data Index: 112557 (C4 (Web Text))

Max Activating Token Index: 883

Text #18

Max Range: 3.4870. Min Range: -3.4870

Max Act: 3.2988. Min Act: -0.1700

Data Index: 1100328 (C4 (Web Text))

Max Activating Token Index: 973

Text #19

Max Range: 3.4870. Min Range: -3.4870

Max Act: 3.2831. Min Act: -0.1700

Data Index: 715179 (C4 (Web Text))

Max Activating Token Index: 608