Model: GELU Model: 2 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 472 in Layer 0

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('gelu-2l')

Text #0

Max Range: 3.1598. Min Range: -3.1598

Max Act: 3.1598. Min Act: -0.1700

Data Index: 201352 (C4 (Web Text))

Max Activating Token Index: 965

Text #1

Max Range: 3.1598. Min Range: -3.1598

Max Act: 3.1125. Min Act: -0.1700

Data Index: 698046 (C4 (Web Text))

Max Activating Token Index: 96

Text #2

Max Range: 3.1598. Min Range: -3.1598

Max Act: 3.0967. Min Act: -0.1700

Data Index: 949216 (C4 (Web Text))

Max Activating Token Index: 975

Text #3

Max Range: 3.1598. Min Range: -3.1598

Max Act: 3.0810. Min Act: -0.1700

Data Index: 349294 (C4 (Web Text))

Max Activating Token Index: 357

Text #4

Max Range: 3.1598. Min Range: -3.1598

Max Act: 3.0652. Min Act: -0.1700

Data Index: 404664 (C4 (Web Text))

Max Activating Token Index: 940

Text #5

Max Range: 3.1598. Min Range: -3.1598

Max Act: 3.0336. Min Act: -0.1700

Data Index: 325058 (C4 (Web Text))

Max Activating Token Index: 1008

Text #6

Max Range: 3.1598. Min Range: -3.1598

Max Act: 3.0178. Min Act: -0.1700

Data Index: 91209 (C4 (Web Text))

Max Activating Token Index: 983

Text #7

Max Range: 3.1598. Min Range: -3.1598

Max Act: 3.0336. Min Act: -0.1700

Data Index: 1148174 (C4 (Web Text))

Max Activating Token Index: 562

Text #8

Max Range: 3.1598. Min Range: -3.1598

Max Act: 2.9862. Min Act: -0.1700

Data Index: 1000067 (C4 (Web Text))

Max Activating Token Index: 302

Text #9

Max Range: 3.1598. Min Range: -3.1598

Max Act: 2.9862. Min Act: -0.1700

Data Index: 1280546 (C4 (Web Text))

Max Activating Token Index: 37

Text #10

Max Range: 3.1598. Min Range: -3.1598

Max Act: 2.9862. Min Act: -0.1700

Data Index: 1189599 (C4 (Web Text))

Max Activating Token Index: 763

Text #11

Max Range: 3.1598. Min Range: -3.1598

Max Act: 2.9387. Min Act: -0.1700

Data Index: 521257 (C4 (Web Text))

Max Activating Token Index: 927

Text #12

Max Range: 3.1598. Min Range: -3.1598

Max Act: 2.9704. Min Act: -0.1700

Data Index: 636757 (C4 (Web Text))

Max Activating Token Index: 665

Text #13

Max Range: 3.1598. Min Range: -3.1598

Max Act: 2.9229. Min Act: -0.1700

Data Index: 1117412 (C4 (Web Text))

Max Activating Token Index: 35

Text #14

Max Range: 3.1598. Min Range: -3.1598

Max Act: 2.9387. Min Act: -0.1699

Data Index: 1570133 (Python Code)

Max Activating Token Index: 534

Text #15

Max Range: 3.1598. Min Range: -3.1598

Max Act: 2.9229. Min Act: -0.1700

Data Index: 455663 (C4 (Web Text))

Max Activating Token Index: 357

Text #16

Max Range: 3.1598. Min Range: -3.1598

Max Act: 2.9229. Min Act: -0.1700

Data Index: 1214917 (C4 (Web Text))

Max Activating Token Index: 549

Text #17

Max Range: 3.1598. Min Range: -3.1598

Max Act: 2.8912. Min Act: -0.1700

Data Index: 936108 (C4 (Web Text))

Max Activating Token Index: 865

Text #18

Max Range: 3.1598. Min Range: -3.1598

Max Act: 2.8912. Min Act: -0.1700

Data Index: 299084 (C4 (Web Text))

Max Activating Token Index: 804

Text #19

Max Range: 3.1598. Min Range: -3.1598

Max Act: 2.9070. Min Act: -0.1700

Data Index: 205152 (C4 (Web Text))

Max Activating Token Index: 856