Model: GELU Model: 4 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1153 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('gelu-4l')

Text #0

Max Range: 3.7768. Min Range: -3.7768

Max Act: 3.7768. Min Act: -0.1700

Data Index: 313345 (C4 (Web Text))

Max Activating Token Index: 699

Text #1

Max Range: 3.7768. Min Range: -3.7768

Max Act: 3.7455. Min Act: -0.1700

Data Index: 1284013 (C4 (Web Text))

Max Activating Token Index: 654

Text #2

Max Range: 3.7768. Min Range: -3.7768

Max Act: 3.6829. Min Act: -0.1700

Data Index: 781626 (C4 (Web Text))

Max Activating Token Index: 209

Text #3

Max Range: 3.7768. Min Range: -3.7768

Max Act: 3.6829. Min Act: -0.1700

Data Index: 947054 (C4 (Web Text))

Max Activating Token Index: 429

Text #4

Max Range: 3.7768. Min Range: -3.7768

Max Act: 3.6829. Min Act: -0.1700

Data Index: 669727 (C4 (Web Text))

Max Activating Token Index: 831

Text #5

Max Range: 3.7768. Min Range: -3.7768

Max Act: 3.6985. Min Act: -0.1700

Data Index: 1053045 (C4 (Web Text))

Max Activating Token Index: 612

Text #6

Max Range: 3.7768. Min Range: -3.7768

Max Act: 3.6829. Min Act: -0.1700

Data Index: 563845 (C4 (Web Text))

Max Activating Token Index: 415

Text #7

Max Range: 3.7768. Min Range: -3.7768

Max Act: 3.6046. Min Act: -0.1700

Data Index: 269229 (C4 (Web Text))

Max Activating Token Index: 679

Text #8

Max Range: 3.7768. Min Range: -3.7768

Max Act: 3.6672. Min Act: -0.1700

Data Index: 142758 (C4 (Web Text))

Max Activating Token Index: 703

Text #9

Max Range: 3.7768. Min Range: -3.7768

Max Act: 3.6203. Min Act: -0.1700

Data Index: 412602 (C4 (Web Text))

Max Activating Token Index: 563

Text #10

Max Range: 3.7768. Min Range: -3.7768

Max Act: 3.6359. Min Act: -0.1700

Data Index: 404333 (C4 (Web Text))

Max Activating Token Index: 884

Text #11

Max Range: 3.7768. Min Range: -3.7768

Max Act: 3.6516. Min Act: -0.1700

Data Index: 1311335 (C4 (Web Text))

Max Activating Token Index: 464

Text #12

Max Range: 3.7768. Min Range: -3.7768

Max Act: 3.6203. Min Act: -0.1700

Data Index: 16281 (C4 (Web Text))

Max Activating Token Index: 924

Text #13

Max Range: 3.7768. Min Range: -3.7768

Max Act: 3.6203. Min Act: -0.1700

Data Index: 441633 (C4 (Web Text))

Max Activating Token Index: 705

Text #14

Max Range: 3.7768. Min Range: -3.7768

Max Act: 3.5890. Min Act: -0.1700

Data Index: 1163700 (C4 (Web Text))

Max Activating Token Index: 766

Text #15

Max Range: 3.7768. Min Range: -3.7768

Max Act: 3.5733. Min Act: -0.1700

Data Index: 1301122 (C4 (Web Text))

Max Activating Token Index: 847

Text #16

Max Range: 3.7768. Min Range: -3.7768

Max Act: 3.5890. Min Act: -0.1700

Data Index: 1139848 (C4 (Web Text))

Max Activating Token Index: 611

Text #17

Max Range: 3.7768. Min Range: -3.7768

Max Act: 3.5890. Min Act: -0.1700

Data Index: 68493 (C4 (Web Text))

Max Activating Token Index: 19

Text #18

Max Range: 3.7768. Min Range: -3.7768

Max Act: 3.5890. Min Act: -0.1700

Data Index: 870922 (C4 (Web Text))

Max Activating Token Index: 720

Text #19

Max Range: 3.7768. Min Range: -3.7768

Max Act: 3.5420. Min Act: -0.1700

Data Index: 925656 (C4 (Web Text))

Max Activating Token Index: 801