Model: GELU Model: 3 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1649 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('gelu-3l')

Text #0

Max Range: 4.1943. Min Range: -4.1943

Max Act: 4.1943. Min Act: -0.1700

Data Index: 1250799 (C4 (Web Text))

Max Activating Token Index: 445

Text #1

Max Range: 4.1943. Min Range: -4.1943

Max Act: 4.0380. Min Act: -0.1700

Data Index: 637581 (C4 (Web Text))

Max Activating Token Index: 170

Text #2

Max Range: 4.1943. Min Range: -4.1943

Max Act: 3.9755. Min Act: -0.1700

Data Index: 1224622 (C4 (Web Text))

Max Activating Token Index: 1023

Text #3

Max Range: 4.1943. Min Range: -4.1943

Max Act: 4.0067. Min Act: -0.1700

Data Index: 1125634 (C4 (Web Text))

Max Activating Token Index: 705

Text #4

Max Range: 4.1943. Min Range: -4.1943

Max Act: 3.8817. Min Act: -0.1700

Data Index: 1264131 (C4 (Web Text))

Max Activating Token Index: 488

Text #5

Max Range: 4.1943. Min Range: -4.1943

Max Act: 3.9129. Min Act: -0.1700

Data Index: 701918 (C4 (Web Text))

Max Activating Token Index: 995

Text #6

Max Range: 4.1943. Min Range: -4.1943

Max Act: 3.9755. Min Act: -0.1700

Data Index: 505635 (C4 (Web Text))

Max Activating Token Index: 643

Text #7

Max Range: 4.1943. Min Range: -4.1943

Max Act: 3.8817. Min Act: -0.1700

Data Index: 995959 (C4 (Web Text))

Max Activating Token Index: 979

Text #8

Max Range: 4.1943. Min Range: -4.1943

Max Act: 3.8817. Min Act: -0.1700

Data Index: 458236 (C4 (Web Text))

Max Activating Token Index: 900

Text #9

Max Range: 4.1943. Min Range: -4.1943

Max Act: 3.9129. Min Act: -0.1700

Data Index: 439625 (C4 (Web Text))

Max Activating Token Index: 100

Text #10

Max Range: 4.1943. Min Range: -4.1943

Max Act: 3.9129. Min Act: -0.1700

Data Index: 26632 (C4 (Web Text))

Max Activating Token Index: 488

Text #11

Max Range: 4.1943. Min Range: -4.1943

Max Act: 3.8191. Min Act: -0.1700

Data Index: 59374 (C4 (Web Text))

Max Activating Token Index: 932

Text #12

Max Range: 4.1943. Min Range: -4.1943

Max Act: 3.9129. Min Act: -0.1700

Data Index: 732273 (C4 (Web Text))

Max Activating Token Index: 546

Text #13

Max Range: 4.1943. Min Range: -4.1943

Max Act: 3.8817. Min Act: -0.1700

Data Index: 527271 (C4 (Web Text))

Max Activating Token Index: 502

Text #14

Max Range: 4.1943. Min Range: -4.1943

Max Act: 3.8817. Min Act: -0.1700

Data Index: 80494 (C4 (Web Text))

Max Activating Token Index: 187

Text #15

Max Range: 4.1943. Min Range: -4.1943

Max Act: 3.8504. Min Act: -0.1700

Data Index: 122785 (C4 (Web Text))

Max Activating Token Index: 734

Text #16

Max Range: 4.1943. Min Range: -4.1943

Max Act: 3.7565. Min Act: -0.1700

Data Index: 1366938 (Python Code)

Max Activating Token Index: 172

Text #17

Max Range: 4.1943. Min Range: -4.1943

Max Act: 3.7878. Min Act: -0.1700

Data Index: 53974 (C4 (Web Text))

Max Activating Token Index: 552

Text #18

Max Range: 4.1943. Min Range: -4.1943

Max Act: 3.8504. Min Act: -0.1700

Data Index: 895802 (C4 (Web Text))

Max Activating Token Index: 974

Text #19

Max Range: 4.1943. Min Range: -4.1943

Max Act: 3.8191. Min Act: -0.1700

Data Index: 836673 (C4 (Web Text))

Max Activating Token Index: 536