Model: GELU Model: 4 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1471 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('gelu-4l')

Text #0

Max Range: 5.5900. Min Range: -5.5900

Max Act: 5.5900. Min Act: -0.1700

Data Index: 1225458 (C4 (Web Text))

Max Activating Token Index: 255

Text #1

Max Range: 5.5900. Min Range: -5.5900

Max Act: 5.5275. Min Act: -0.1700

Data Index: 362366 (C4 (Web Text))

Max Activating Token Index: 310

Text #2

Max Range: 5.5900. Min Range: -5.5900

Max Act: 5.3713. Min Act: -0.1700

Data Index: 712984 (C4 (Web Text))

Max Activating Token Index: 815

Text #3

Max Range: 5.5900. Min Range: -5.5900

Max Act: 5.3088. Min Act: -0.1700

Data Index: 1349441 (C4 (Web Text))

Max Activating Token Index: 613

Text #4

Max Range: 5.5900. Min Range: -5.5900

Max Act: 5.4338. Min Act: -0.1700

Data Index: 761045 (C4 (Web Text))

Max Activating Token Index: 520

Text #5

Max Range: 5.5900. Min Range: -5.5900

Max Act: 5.4025. Min Act: -0.1700

Data Index: 350235 (C4 (Web Text))

Max Activating Token Index: 896

Text #6

Max Range: 5.5900. Min Range: -5.5900

Max Act: 5.4025. Min Act: -0.1700

Data Index: 422650 (C4 (Web Text))

Max Activating Token Index: 540

Text #7

Max Range: 5.5900. Min Range: -5.5900

Max Act: 5.3713. Min Act: -0.1700

Data Index: 1014094 (C4 (Web Text))

Max Activating Token Index: 996

Text #8

Max Range: 5.5900. Min Range: -5.5900

Max Act: 5.3400. Min Act: -0.1700

Data Index: 627441 (C4 (Web Text))

Max Activating Token Index: 948

Text #9

Max Range: 5.5900. Min Range: -5.5900

Max Act: 5.3713. Min Act: -0.1700

Data Index: 873362 (C4 (Web Text))

Max Activating Token Index: 951

Text #10

Max Range: 5.5900. Min Range: -5.5900

Max Act: 5.4025. Min Act: -0.1700

Data Index: 919759 (C4 (Web Text))

Max Activating Token Index: 68

Text #11

Max Range: 5.5900. Min Range: -5.5900

Max Act: 5.3400. Min Act: -0.1700

Data Index: 1298653 (C4 (Web Text))

Max Activating Token Index: 683

Text #12

Max Range: 5.5900. Min Range: -5.5900

Max Act: 5.3400. Min Act: -0.1700

Data Index: 780958 (C4 (Web Text))

Max Activating Token Index: 923

Text #13

Max Range: 5.5900. Min Range: -5.5900

Max Act: 5.3088. Min Act: -0.1700

Data Index: 1064130 (C4 (Web Text))

Max Activating Token Index: 286

Text #14

Max Range: 5.5900. Min Range: -5.5900

Max Act: 5.3400. Min Act: -0.1700

Data Index: 965920 (C4 (Web Text))

Max Activating Token Index: 167

Text #15

Max Range: 5.5900. Min Range: -5.5900

Max Act: 5.3088. Min Act: -0.1700

Data Index: 1229464 (C4 (Web Text))

Max Activating Token Index: 183

Text #16

Max Range: 5.5900. Min Range: -5.5900

Max Act: 5.3713. Min Act: -0.1700

Data Index: 129925 (C4 (Web Text))

Max Activating Token Index: 665

Text #17

Max Range: 5.5900. Min Range: -5.5900

Max Act: 5.3713. Min Act: -0.1700

Data Index: 1343682 (C4 (Web Text))

Max Activating Token Index: 835

Text #18

Max Range: 5.5900. Min Range: -5.5900

Max Act: 5.2775. Min Act: -0.1700

Data Index: 1221407 (C4 (Web Text))

Max Activating Token Index: 887

Text #19

Max Range: 5.5900. Min Range: -5.5900

Max Act: 5.3400. Min Act: -0.1700

Data Index: 1268092 (C4 (Web Text))

Max Activating Token Index: 813