Model: GELU Model: 4 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 943 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('gelu-4l')

Text #0

Max Range: 4.1513. Min Range: -4.1513

Max Act: 4.1513. Min Act: -0.1700

Data Index: 665992 (C4 (Web Text))

Max Activating Token Index: 349

Text #1

Max Range: 4.1513. Min Range: -4.1513

Max Act: 4.0888. Min Act: -0.1700

Data Index: 428334 (C4 (Web Text))

Max Activating Token Index: 685

Text #2

Max Range: 4.1513. Min Range: -4.1513

Max Act: 4.0888. Min Act: -0.1700

Data Index: 1110265 (C4 (Web Text))

Max Activating Token Index: 383

Text #3

Max Range: 4.1513. Min Range: -4.1513

Max Act: 4.0263. Min Act: -0.1700

Data Index: 1207554 (C4 (Web Text))

Max Activating Token Index: 958

Text #4

Max Range: 4.1513. Min Range: -4.1513

Max Act: 4.0263. Min Act: -0.1700

Data Index: 172090 (C4 (Web Text))

Max Activating Token Index: 243

Text #5

Max Range: 4.1513. Min Range: -4.1513

Max Act: 3.9950. Min Act: -0.1700

Data Index: 1128585 (C4 (Web Text))

Max Activating Token Index: 768

Text #6

Max Range: 4.1513. Min Range: -4.1513

Max Act: 3.9950. Min Act: -0.1700

Data Index: 717879 (C4 (Web Text))

Max Activating Token Index: 552

Text #7

Max Range: 4.1513. Min Range: -4.1513

Max Act: 3.9325. Min Act: -0.1700

Data Index: 1249329 (C4 (Web Text))

Max Activating Token Index: 860

Text #8

Max Range: 4.1513. Min Range: -4.1513

Max Act: 3.9325. Min Act: -0.1700

Data Index: 419835 (C4 (Web Text))

Max Activating Token Index: 504

Text #9

Max Range: 4.1513. Min Range: -4.1513

Max Act: 3.9325. Min Act: -0.1700

Data Index: 1167788 (C4 (Web Text))

Max Activating Token Index: 253

Text #10

Max Range: 4.1513. Min Range: -4.1513

Max Act: 3.9950. Min Act: -0.1700

Data Index: 1242374 (C4 (Web Text))

Max Activating Token Index: 566

Text #11

Max Range: 4.1513. Min Range: -4.1513

Max Act: 3.9638. Min Act: -0.1700

Data Index: 1261772 (C4 (Web Text))

Max Activating Token Index: 299

Text #12

Max Range: 4.1513. Min Range: -4.1513

Max Act: 3.9638. Min Act: -0.1700

Data Index: 440768 (C4 (Web Text))

Max Activating Token Index: 852

Text #13

Max Range: 4.1513. Min Range: -4.1513

Max Act: 3.9325. Min Act: -0.1700

Data Index: 350883 (C4 (Web Text))

Max Activating Token Index: 398

Text #14

Max Range: 4.1513. Min Range: -4.1513

Max Act: 3.9012. Min Act: -0.1700

Data Index: 539063 (C4 (Web Text))

Max Activating Token Index: 198

Text #15

Max Range: 4.1513. Min Range: -4.1513

Max Act: 3.9012. Min Act: -0.1700

Data Index: 108085 (C4 (Web Text))

Max Activating Token Index: 255

Text #16

Max Range: 4.1513. Min Range: -4.1513

Max Act: 3.9012. Min Act: -0.1700

Data Index: 138358 (C4 (Web Text))

Max Activating Token Index: 424

Text #17

Max Range: 4.1513. Min Range: -4.1513

Max Act: 3.8700. Min Act: -0.1700

Data Index: 1098557 (C4 (Web Text))

Max Activating Token Index: 425

Text #18

Max Range: 4.1513. Min Range: -4.1513

Max Act: 3.9012. Min Act: -0.1700

Data Index: 842463 (C4 (Web Text))

Max Activating Token Index: 383

Text #19

Max Range: 4.1513. Min Range: -4.1513

Max Act: 3.8856. Min Act: -0.1700

Data Index: 108807 (C4 (Web Text))

Max Activating Token Index: 539