Model: GELU Model: 3 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 439 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('gelu-3l')

Text #0

Max Range: 7.4383. Min Range: -7.4383

Max Act: 7.4383. Min Act: -0.1700

Data Index: 812866 (C4 (Web Text))

Max Activating Token Index: 488

Text #1

Max Range: 7.4383. Min Range: -7.4383

Max Act: 7.2508. Min Act: -0.1700

Data Index: 1471900 (Python Code)

Max Activating Token Index: 412

Text #2

Max Range: 7.4383. Min Range: -7.4383

Max Act: 7.1883. Min Act: -0.1700

Data Index: 1067745 (C4 (Web Text))

Max Activating Token Index: 742

Text #3

Max Range: 7.4383. Min Range: -7.4383

Max Act: 7.2508. Min Act: -0.1700

Data Index: 1552599 (Python Code)

Max Activating Token Index: 178

Text #4

Max Range: 7.4383. Min Range: -7.4383

Max Act: 7.1258. Min Act: -0.1700

Data Index: 1475786 (Python Code)

Max Activating Token Index: 544

Text #5

Max Range: 7.4383. Min Range: -7.4383

Max Act: 7.0946. Min Act: -0.1700

Data Index: 1487313 (Python Code)

Max Activating Token Index: 407

Text #6

Max Range: 7.4383. Min Range: -7.4383

Max Act: 7.1571. Min Act: -0.1700

Data Index: 1385831 (Python Code)

Max Activating Token Index: 293

Text #7

Max Range: 7.4383. Min Range: -7.4383

Max Act: 7.0633. Min Act: -0.1700

Data Index: 1182616 (C4 (Web Text))

Max Activating Token Index: 579

Text #8

Max Range: 7.4383. Min Range: -7.4383

Max Act: 7.1571. Min Act: -0.1700

Data Index: 1613270 (Python Code)

Max Activating Token Index: 767

Text #9

Max Range: 7.4383. Min Range: -7.4383

Max Act: 7.1258. Min Act: -0.1700

Data Index: 1510537 (Python Code)

Max Activating Token Index: 75

Text #10

Max Range: 7.4383. Min Range: -7.4383

Max Act: 7.0946. Min Act: -0.1700

Data Index: 1528574 (Python Code)

Max Activating Token Index: 248

Text #11

Max Range: 7.4383. Min Range: -7.4383

Max Act: 7.0946. Min Act: -0.1700

Data Index: 1394205 (Python Code)

Max Activating Token Index: 530

Text #12

Max Range: 7.4383. Min Range: -7.4383

Max Act: 7.1258. Min Act: -0.1700

Data Index: 1430484 (Python Code)

Max Activating Token Index: 713

Text #13

Max Range: 7.4383. Min Range: -7.4383

Max Act: 7.0633. Min Act: -0.1700

Data Index: 1554064 (Python Code)

Max Activating Token Index: 512

Text #14

Max Range: 7.4383. Min Range: -7.4383

Max Act: 7.0321. Min Act: -0.1700

Data Index: 1522088 (Python Code)

Max Activating Token Index: 687

Text #15

Max Range: 7.4383. Min Range: -7.4383

Max Act: 7.0008. Min Act: -0.1700

Data Index: 1416571 (Python Code)

Max Activating Token Index: 738

Text #16

Max Range: 7.4383. Min Range: -7.4383

Max Act: 7.0633. Min Act: -0.1700

Data Index: 1551423 (Python Code)

Max Activating Token Index: 312

Text #17

Max Range: 7.4383. Min Range: -7.4383

Max Act: 7.0633. Min Act: -0.1700

Data Index: 1454927 (Python Code)

Max Activating Token Index: 922

Text #18

Max Range: 7.4383. Min Range: -7.4383

Max Act: 6.9383. Min Act: -0.1700

Data Index: 1387685 (Python Code)

Max Activating Token Index: 836

Text #19

Max Range: 7.4383. Min Range: -7.4383

Max Act: 7.0633. Min Act: -0.1700

Data Index: 141312 (C4 (Web Text))

Max Activating Token Index: 1012