Model: GELU Model: 4 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1431 in Layer 3

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('gelu-4l')

Text #0

Max Range: 5.0832. Min Range: -5.0832

Max Act: 5.0832. Min Act: -0.1700

Data Index: 969474 (C4 (Web Text))

Max Activating Token Index: 101

Text #1

Max Range: 5.0832. Min Range: -5.0832

Max Act: 4.8957. Min Act: -0.1700

Data Index: 409200 (C4 (Web Text))

Max Activating Token Index: 283

Text #2

Max Range: 5.0832. Min Range: -5.0832

Max Act: 4.8332. Min Act: -0.1700

Data Index: 93801 (C4 (Web Text))

Max Activating Token Index: 676

Text #3

Max Range: 5.0832. Min Range: -5.0832

Max Act: 4.8488. Min Act: -0.1700

Data Index: 782979 (C4 (Web Text))

Max Activating Token Index: 300

Text #4

Max Range: 5.0832. Min Range: -5.0832

Max Act: 4.8020. Min Act: -0.1700

Data Index: 514883 (C4 (Web Text))

Max Activating Token Index: 958

Text #5

Max Range: 5.0832. Min Range: -5.0832

Max Act: 4.7395. Min Act: -0.1700

Data Index: 133761 (C4 (Web Text))

Max Activating Token Index: 398

Text #6

Max Range: 5.0832. Min Range: -5.0832

Max Act: 4.6613. Min Act: -0.1700

Data Index: 927942 (C4 (Web Text))

Max Activating Token Index: 263

Text #7

Max Range: 5.0832. Min Range: -5.0832

Max Act: 4.7238. Min Act: -0.1700

Data Index: 795151 (C4 (Web Text))

Max Activating Token Index: 858

Text #8

Max Range: 5.0832. Min Range: -5.0832

Max Act: 4.6301. Min Act: -0.1700

Data Index: 653844 (C4 (Web Text))

Max Activating Token Index: 438

Text #9

Max Range: 5.0832. Min Range: -5.0832

Max Act: 4.6613. Min Act: -0.1700

Data Index: 524888 (C4 (Web Text))

Max Activating Token Index: 909

Text #10

Max Range: 5.0832. Min Range: -5.0832

Max Act: 4.7082. Min Act: -0.1700

Data Index: 387542 (C4 (Web Text))

Max Activating Token Index: 859

Text #11

Max Range: 5.0832. Min Range: -5.0832

Max Act: 4.6613. Min Act: -0.1700

Data Index: 1297543 (C4 (Web Text))

Max Activating Token Index: 822

Text #12

Max Range: 5.0832. Min Range: -5.0832

Max Act: 4.5988. Min Act: -0.1700

Data Index: 668448 (C4 (Web Text))

Max Activating Token Index: 812

Text #13

Max Range: 5.0832. Min Range: -5.0832

Max Act: 4.6457. Min Act: -0.1700

Data Index: 643835 (C4 (Web Text))

Max Activating Token Index: 438

Text #14

Max Range: 5.0832. Min Range: -5.0832

Max Act: 4.5988. Min Act: -0.1700

Data Index: 489051 (C4 (Web Text))

Max Activating Token Index: 945

Text #15

Max Range: 5.0832. Min Range: -5.0832

Max Act: 4.5832. Min Act: -0.1700

Data Index: 816807 (C4 (Web Text))

Max Activating Token Index: 955

Text #16

Max Range: 5.0832. Min Range: -5.0832

Max Act: 4.5520. Min Act: -0.1700

Data Index: 1101321 (C4 (Web Text))

Max Activating Token Index: 489

Text #17

Max Range: 5.0832. Min Range: -5.0832

Max Act: 4.5676. Min Act: -0.1700

Data Index: 896500 (C4 (Web Text))

Max Activating Token Index: 513

Text #18

Max Range: 5.0832. Min Range: -5.0832

Max Act: 4.5832. Min Act: -0.1700

Data Index: 718235 (C4 (Web Text))

Max Activating Token Index: 360

Text #19

Max Range: 5.0832. Min Range: -5.0832

Max Act: 4.5051. Min Act: -0.1700

Data Index: 708272 (C4 (Web Text))

Max Activating Token Index: 922