Model: GELU Model: 1 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1950 in Layer 0

Load this data into an Interactive Neuroscope

See Documentation here

Hooked Transformer Loading: HookedTransformer.from_pretrained('gelu-1l')

Click the toggle to see the full text!

Text #0

Max Range: 4.1454. Min Range: -4.1454

Max Act: 4.1454. Min Act: -0.1685

Data Index: 180001 (C4 (Web Text))

Max Activating Token Index: 339

Text #1

Max Range: 4.1454. Min Range: -4.1454

Max Act: 4.1251. Min Act: -0.1700

Data Index: 720259 (C4 (Web Text))

Max Activating Token Index: 948

Text #2

Max Range: 4.1454. Min Range: -4.1454

Max Act: 4.0388. Min Act: -0.1700

Data Index: 1561412 (Python Code)

Max Activating Token Index: 833

Text #3

Max Range: 4.1454. Min Range: -4.1454

Max Act: 4.0031. Min Act: -0.1700

Data Index: 1472949 (Python Code)

Max Activating Token Index: 166

Text #4

Max Range: 4.1454. Min Range: -4.1454

Max Act: 3.9366. Min Act: -0.1700

Data Index: 1518718 (Python Code)

Max Activating Token Index: 834

Text #5

Max Range: 4.1454. Min Range: -4.1454

Max Act: 3.9075. Min Act: -0.1700

Data Index: 1249237 (C4 (Web Text))

Max Activating Token Index: 675

Text #6

Max Range: 4.1454. Min Range: -4.1454

Max Act: 3.9152. Min Act: -0.1699

Data Index: 1469261 (Python Code)

Max Activating Token Index: 763

Text #7

Max Range: 4.1454. Min Range: -4.1454

Max Act: 3.8918. Min Act: -0.1697

Data Index: 563396 (C4 (Web Text))

Max Activating Token Index: 526

Text #8

Max Range: 4.1454. Min Range: -4.1454

Max Act: 3.8971. Min Act: -0.1700

Data Index: 36202 (C4 (Web Text))

Max Activating Token Index: 824

Text #9

Max Range: 4.1454. Min Range: -4.1454

Max Act: 3.9020. Min Act: -0.1699

Data Index: 316941 (C4 (Web Text))

Max Activating Token Index: 559

Text #10

Max Range: 4.1454. Min Range: -4.1454

Max Act: 3.8925. Min Act: -0.1699

Data Index: 1193477 (C4 (Web Text))

Max Activating Token Index: 1023

Text #11

Max Range: 4.1454. Min Range: -4.1454

Max Act: 3.8633. Min Act: -0.1700

Data Index: 935991 (C4 (Web Text))

Max Activating Token Index: 527

Text #12

Max Range: 4.1454. Min Range: -4.1454

Max Act: 3.8705. Min Act: -0.1644

Data Index: 1655959 (Python Code)

Max Activating Token Index: 551

Text #13

Max Range: 4.1454. Min Range: -4.1454

Max Act: 3.8596. Min Act: -0.1446

Data Index: 1555506 (Python Code)

Max Activating Token Index: 530

Text #14

Max Range: 4.1454. Min Range: -4.1454

Max Act: 3.8573. Min Act: -0.1700

Data Index: 458527 (C4 (Web Text))

Max Activating Token Index: 886

Text #15

Max Range: 4.1454. Min Range: -4.1454

Max Act: 3.8373. Min Act: -0.1687

Data Index: 802737 (C4 (Web Text))

Max Activating Token Index: 472

Text #16

Max Range: 4.1454. Min Range: -4.1454

Max Act: 3.8442. Min Act: -0.1697

Data Index: 1645194 (Python Code)

Max Activating Token Index: 765

Text #17

Max Range: 4.1454. Min Range: -4.1454

Max Act: 3.8292. Min Act: -0.1699

Data Index: 1652148 (Python Code)

Max Activating Token Index: 1020

Text #18

Max Range: 4.1454. Min Range: -4.1454

Max Act: 3.8235. Min Act: -0.1700

Data Index: 345352 (C4 (Web Text))

Max Activating Token Index: 414

Text #19

Max Range: 4.1454. Min Range: -4.1454

Max Act: 3.7792. Min Act: -0.1698

Data Index: 88562 (C4 (Web Text))

Max Activating Token Index: 854