Model: GELU Model: 4 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1765 in Layer 1

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('gelu-4l')

Text #0

Max Range: 5.0256. Min Range: -5.0256

Max Act: 5.0256. Min Act: -0.1700

Data Index: 103261 (C4 (Web Text))

Max Activating Token Index: 620

Text #1

Max Range: 5.0256. Min Range: -5.0256

Max Act: 4.9631. Min Act: -0.1700

Data Index: 719319 (C4 (Web Text))

Max Activating Token Index: 332

Text #2

Max Range: 5.0256. Min Range: -5.0256

Max Act: 4.9318. Min Act: -0.1700

Data Index: 641180 (C4 (Web Text))

Max Activating Token Index: 907

Text #3

Max Range: 5.0256. Min Range: -5.0256

Max Act: 4.8693. Min Act: -0.1700

Data Index: 1000048 (C4 (Web Text))

Max Activating Token Index: 684

Text #4

Max Range: 5.0256. Min Range: -5.0256

Max Act: 4.8068. Min Act: -0.1700

Data Index: 1051037 (C4 (Web Text))

Max Activating Token Index: 912

Text #5

Max Range: 5.0256. Min Range: -5.0256

Max Act: 4.8381. Min Act: -0.1699

Data Index: 1235783 (C4 (Web Text))

Max Activating Token Index: 942

Text #6

Max Range: 5.0256. Min Range: -5.0256

Max Act: 4.8381. Min Act: -0.1700

Data Index: 164154 (C4 (Web Text))

Max Activating Token Index: 159

Text #7

Max Range: 5.0256. Min Range: -5.0256

Max Act: 4.8381. Min Act: -0.1700

Data Index: 519814 (C4 (Web Text))

Max Activating Token Index: 761

Text #8

Max Range: 5.0256. Min Range: -5.0256

Max Act: 4.7756. Min Act: -0.1700

Data Index: 489045 (C4 (Web Text))

Max Activating Token Index: 427

Text #9

Max Range: 5.0256. Min Range: -5.0256

Max Act: 4.7756. Min Act: -0.1699

Data Index: 460864 (C4 (Web Text))

Max Activating Token Index: 65

Text #10

Max Range: 5.0256. Min Range: -5.0256

Max Act: 4.8381. Min Act: -0.1700

Data Index: 696780 (C4 (Web Text))

Max Activating Token Index: 342

Text #11

Max Range: 5.0256. Min Range: -5.0256

Max Act: 4.7756. Min Act: -0.1700

Data Index: 818489 (C4 (Web Text))

Max Activating Token Index: 86

Text #12

Max Range: 5.0256. Min Range: -5.0256

Max Act: 4.7443. Min Act: -0.1700

Data Index: 459105 (C4 (Web Text))

Max Activating Token Index: 758

Text #13

Max Range: 5.0256. Min Range: -5.0256

Max Act: 4.7756. Min Act: -0.1700

Data Index: 57434 (C4 (Web Text))

Max Activating Token Index: 879

Text #14

Max Range: 5.0256. Min Range: -5.0256

Max Act: 4.8068. Min Act: -0.1700

Data Index: 1332778 (C4 (Web Text))

Max Activating Token Index: 526

Text #15

Max Range: 5.0256. Min Range: -5.0256

Max Act: 4.7443. Min Act: -0.1700

Data Index: 488477 (C4 (Web Text))

Max Activating Token Index: 273

Text #16

Max Range: 5.0256. Min Range: -5.0256

Max Act: 4.7443. Min Act: -0.1700

Data Index: 1228773 (C4 (Web Text))

Max Activating Token Index: 733

Text #17

Max Range: 5.0256. Min Range: -5.0256

Max Act: 4.7443. Min Act: -0.1700

Data Index: 927371 (C4 (Web Text))

Max Activating Token Index: 619

Text #18

Max Range: 5.0256. Min Range: -5.0256

Max Act: 4.7131. Min Act: -0.1699

Data Index: 554867 (C4 (Web Text))

Max Activating Token Index: 302

Text #19

Max Range: 5.0256. Min Range: -5.0256

Max Act: 4.7131. Min Act: -0.1700

Data Index: 976244 (C4 (Web Text))

Max Activating Token Index: 970