Model: GELU Model: 4 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1393 in Layer 3

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('gelu-4l')

Text #0

Max Range: 8.5340. Min Range: -8.5340

Max Act: 8.5340. Min Act: -0.1700

Data Index: 1597763 (Python Code)

Max Activating Token Index: 417

Text #1

Max Range: 8.5340. Min Range: -8.5340

Max Act: 8.3778. Min Act: -0.1700

Data Index: 713898 (C4 (Web Text))

Max Activating Token Index: 256

Text #2

Max Range: 8.5340. Min Range: -8.5340

Max Act: 8.3778. Min Act: -0.1700

Data Index: 136737 (C4 (Web Text))

Max Activating Token Index: 321

Text #3

Max Range: 8.5340. Min Range: -8.5340

Max Act: 8.2528. Min Act: -0.1700

Data Index: 796802 (C4 (Web Text))

Max Activating Token Index: 906

Text #4

Max Range: 8.5340. Min Range: -8.5340

Max Act: 8.2215. Min Act: -0.1700

Data Index: 1051813 (C4 (Web Text))

Max Activating Token Index: 922

Text #5

Max Range: 8.5340. Min Range: -8.5340

Max Act: 8.1903. Min Act: -0.1700

Data Index: 127009 (C4 (Web Text))

Max Activating Token Index: 982

Text #6

Max Range: 8.5340. Min Range: -8.5340

Max Act: 8.0965. Min Act: -0.1700

Data Index: 326960 (C4 (Web Text))

Max Activating Token Index: 278

Text #7

Max Range: 8.5340. Min Range: -8.5340

Max Act: 8.1278. Min Act: -0.1700

Data Index: 14049 (C4 (Web Text))

Max Activating Token Index: 232

Text #8

Max Range: 8.5340. Min Range: -8.5340

Max Act: 8.0028. Min Act: -0.1700

Data Index: 640263 (C4 (Web Text))

Max Activating Token Index: 863

Text #9

Max Range: 8.5340. Min Range: -8.5340

Max Act: 8.0340. Min Act: -0.1700

Data Index: 353632 (C4 (Web Text))

Max Activating Token Index: 403

Text #10

Max Range: 8.5340. Min Range: -8.5340

Max Act: 8.0340. Min Act: -0.1700

Data Index: 365591 (C4 (Web Text))

Max Activating Token Index: 630

Text #11

Max Range: 8.5340. Min Range: -8.5340

Max Act: 7.9403. Min Act: -0.1700

Data Index: 313831 (C4 (Web Text))

Max Activating Token Index: 437

Text #12

Max Range: 8.5340. Min Range: -8.5340

Max Act: 7.9090. Min Act: -0.1700

Data Index: 515770 (C4 (Web Text))

Max Activating Token Index: 789

Text #13

Max Range: 8.5340. Min Range: -8.5340

Max Act: 7.8153. Min Act: -0.1700

Data Index: 556269 (C4 (Web Text))

Max Activating Token Index: 974

Text #14

Max Range: 8.5340. Min Range: -8.5340

Max Act: 7.8153. Min Act: -0.1700

Data Index: 806218 (C4 (Web Text))

Max Activating Token Index: 818

Text #15

Max Range: 8.5340. Min Range: -8.5340

Max Act: 7.8153. Min Act: -0.1700

Data Index: 824461 (C4 (Web Text))

Max Activating Token Index: 106

Text #16

Max Range: 8.5340. Min Range: -8.5340

Max Act: 7.8778. Min Act: -0.1700

Data Index: 1355328 (C4 (Web Text))

Max Activating Token Index: 992

Text #17

Max Range: 8.5340. Min Range: -8.5340

Max Act: 7.7528. Min Act: -0.1700

Data Index: 947769 (C4 (Web Text))

Max Activating Token Index: 394

Text #18

Max Range: 8.5340. Min Range: -8.5340

Max Act: 7.7215. Min Act: -0.1700

Data Index: 1230455 (C4 (Web Text))

Max Activating Token Index: 732

Text #19

Max Range: 8.5340. Min Range: -8.5340

Max Act: 7.8153. Min Act: -0.1700

Data Index: 145932 (C4 (Web Text))

Max Activating Token Index: 575