Model: GELU Model: 4 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1574 in Layer 3

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('gelu-4l')

Text #0

Max Range: 8.0514. Min Range: -8.0514

Max Act: 7.9889. Min Act: -0.1700

Data Index: 1531119 (Python Code)

Max Activating Token Index: 17

Text #1

Max Range: 8.0514. Min Range: -8.0514

Max Act: 8.0514. Min Act: -0.1700

Data Index: 1518685 (Python Code)

Max Activating Token Index: 815

Text #2

Max Range: 8.0514. Min Range: -8.0514

Max Act: 7.8951. Min Act: -0.1700

Data Index: 1396330 (Python Code)

Max Activating Token Index: 93

Text #3

Max Range: 8.0514. Min Range: -8.0514

Max Act: 7.8639. Min Act: -0.1700

Data Index: 1411918 (Python Code)

Max Activating Token Index: 11

Text #4

Max Range: 8.0514. Min Range: -8.0514

Max Act: 7.8014. Min Act: -0.1700

Data Index: 1464350 (Python Code)

Max Activating Token Index: 306

Text #5

Max Range: 8.0514. Min Range: -8.0514

Max Act: 7.8326. Min Act: -0.1700

Data Index: 1465529 (Python Code)

Max Activating Token Index: 110

Text #6

Max Range: 8.0514. Min Range: -8.0514

Max Act: 7.8639. Min Act: -0.1700

Data Index: 1385373 (Python Code)

Max Activating Token Index: 515

Text #7

Max Range: 8.0514. Min Range: -8.0514

Max Act: 7.7701. Min Act: -0.1700

Data Index: 1547180 (Python Code)

Max Activating Token Index: 86

Text #8

Max Range: 8.0514. Min Range: -8.0514

Max Act: 7.8639. Min Act: -0.1700

Data Index: 1378310 (Python Code)

Max Activating Token Index: 55

Text #9

Max Range: 8.0514. Min Range: -8.0514

Max Act: 7.7389. Min Act: -0.1700

Data Index: 1573304 (Python Code)

Max Activating Token Index: 1010

Text #10

Max Range: 8.0514. Min Range: -8.0514

Max Act: 7.7076. Min Act: -0.1700

Data Index: 1403310 (Python Code)

Max Activating Token Index: 133

Text #11

Max Range: 8.0514. Min Range: -8.0514

Max Act: 7.8326. Min Act: -0.1700

Data Index: 1365283 (Python Code)

Max Activating Token Index: 409

Text #12

Max Range: 8.0514. Min Range: -8.0514

Max Act: 7.7701. Min Act: -0.1700

Data Index: 1373493 (Python Code)

Max Activating Token Index: 40

Text #13

Max Range: 8.0514. Min Range: -8.0514

Max Act: 7.5826. Min Act: -0.1700

Data Index: 1458227 (Python Code)

Max Activating Token Index: 18

Text #14

Max Range: 8.0514. Min Range: -8.0514

Max Act: 7.6764. Min Act: -0.1700

Data Index: 1628079 (Python Code)

Max Activating Token Index: 108

Text #15

Max Range: 8.0514. Min Range: -8.0514

Max Act: 7.6764. Min Act: -0.1700

Data Index: 1581829 (Python Code)

Max Activating Token Index: 14

Text #16

Max Range: 8.0514. Min Range: -8.0514

Max Act: 7.7389. Min Act: -0.1700

Data Index: 1617472 (Python Code)

Max Activating Token Index: 96

Text #17

Max Range: 8.0514. Min Range: -8.0514

Max Act: 7.6139. Min Act: -0.1700

Data Index: 1510612 (Python Code)

Max Activating Token Index: 22

Text #18

Max Range: 8.0514. Min Range: -8.0514

Max Act: 7.6139. Min Act: -0.1700

Data Index: 1615443 (Python Code)

Max Activating Token Index: 18

Text #19

Max Range: 8.0514. Min Range: -8.0514

Max Act: 7.5826. Min Act: -0.1700

Data Index: 1559097 (Python Code)

Max Activating Token Index: 5