Model: GELU Model: 3 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 828 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('gelu-3l')

Text #0

Max Range: 12.7431. Min Range: -12.7431

Max Act: 12.7431. Min Act: -0.1700

Data Index: 367232 (C4 (Web Text))

Max Activating Token Index: 829

Text #1

Max Range: 12.7431. Min Range: -12.7431

Max Act: 12.5556. Min Act: -0.1700

Data Index: 554496 (C4 (Web Text))

Max Activating Token Index: 903

Text #2

Max Range: 12.7431. Min Range: -12.7431

Max Act: 12.3681. Min Act: -0.1700

Data Index: 375330 (C4 (Web Text))

Max Activating Token Index: 730

Text #3

Max Range: 12.7431. Min Range: -12.7431

Max Act: 12.3056. Min Act: -0.1700

Data Index: 996250 (C4 (Web Text))

Max Activating Token Index: 965

Text #4

Max Range: 12.7431. Min Range: -12.7431

Max Act: 12.3681. Min Act: -0.1700

Data Index: 1283087 (C4 (Web Text))

Max Activating Token Index: 997

Text #5

Max Range: 12.7431. Min Range: -12.7431

Max Act: 12.1181. Min Act: -0.1700

Data Index: 1195622 (C4 (Web Text))

Max Activating Token Index: 808

Text #6

Max Range: 12.7431. Min Range: -12.7431

Max Act: 11.8681. Min Act: -0.1700

Data Index: 1313306 (C4 (Web Text))

Max Activating Token Index: 902

Text #7

Max Range: 12.7431. Min Range: -12.7431

Max Act: 11.9931. Min Act: -0.1700

Data Index: 235011 (C4 (Web Text))

Max Activating Token Index: 1019

Text #8

Max Range: 12.7431. Min Range: -12.7431

Max Act: 12.1181. Min Act: -0.1700

Data Index: 1075660 (C4 (Web Text))

Max Activating Token Index: 870

Text #9

Max Range: 12.7431. Min Range: -12.7431

Max Act: 11.8681. Min Act: -0.1700

Data Index: 657405 (C4 (Web Text))

Max Activating Token Index: 1020

Text #10

Max Range: 12.7431. Min Range: -12.7431

Max Act: 11.8681. Min Act: -0.1700

Data Index: 775078 (C4 (Web Text))

Max Activating Token Index: 760

Text #11

Max Range: 12.7431. Min Range: -12.7431

Max Act: 11.9306. Min Act: -0.1700

Data Index: 1207757 (C4 (Web Text))

Max Activating Token Index: 940

Text #12

Max Range: 12.7431. Min Range: -12.7431

Max Act: 11.7431. Min Act: -0.1699

Data Index: 1558279 (Python Code)

Max Activating Token Index: 725

Text #13

Max Range: 12.7431. Min Range: -12.7431

Max Act: 12.0556. Min Act: -0.1700

Data Index: 942655 (C4 (Web Text))

Max Activating Token Index: 888

Text #14

Max Range: 12.7431. Min Range: -12.7431

Max Act: 11.6181. Min Act: -0.1700

Data Index: 334878 (C4 (Web Text))

Max Activating Token Index: 692

Text #15

Max Range: 12.7431. Min Range: -12.7431

Max Act: 11.8056. Min Act: -0.1700

Data Index: 632382 (C4 (Web Text))

Max Activating Token Index: 162

Text #16

Max Range: 12.7431. Min Range: -12.7431

Max Act: 11.7431. Min Act: -0.1700

Data Index: 149785 (C4 (Web Text))

Max Activating Token Index: 856

Text #17

Max Range: 12.7431. Min Range: -12.7431

Max Act: 11.6181. Min Act: -0.1700

Data Index: 986592 (C4 (Web Text))

Max Activating Token Index: 7

Text #18

Max Range: 12.7431. Min Range: -12.7431

Max Act: 11.7431. Min Act: -0.1688

Data Index: 1505745 (Python Code)

Max Activating Token Index: 142

Text #19

Max Range: 12.7431. Min Range: -12.7431

Max Act: 11.6181. Min Act: -0.1700

Data Index: 895587 (C4 (Web Text))

Max Activating Token Index: 238