Model: GELU Model: 4 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1761 in Layer 3

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('gelu-4l')

Text #0

Max Range: 8.3760. Min Range: -8.3760

Max Act: 8.3760. Min Act: -0.1700

Data Index: 315637 (C4 (Web Text))

Max Activating Token Index: 340

Text #1

Max Range: 8.3760. Min Range: -8.3760

Max Act: 8.1260. Min Act: -0.1700

Data Index: 883162 (C4 (Web Text))

Max Activating Token Index: 761

Text #2

Max Range: 8.3760. Min Range: -8.3760

Max Act: 8.0010. Min Act: -0.1700

Data Index: 576083 (C4 (Web Text))

Max Activating Token Index: 265

Text #3

Max Range: 8.3760. Min Range: -8.3760

Max Act: 8.0635. Min Act: -0.1700

Data Index: 484762 (C4 (Web Text))

Max Activating Token Index: 923

Text #4

Max Range: 8.3760. Min Range: -8.3760

Max Act: 7.9073. Min Act: -0.1700

Data Index: 176256 (C4 (Web Text))

Max Activating Token Index: 382

Text #5

Max Range: 8.3760. Min Range: -8.3760

Max Act: 7.9385. Min Act: -0.1700

Data Index: 1096819 (C4 (Web Text))

Max Activating Token Index: 1014

Text #6

Max Range: 8.3760. Min Range: -8.3760

Max Act: 7.9073. Min Act: -0.1700

Data Index: 344801 (C4 (Web Text))

Max Activating Token Index: 847

Text #7

Max Range: 8.3760. Min Range: -8.3760

Max Act: 7.9073. Min Act: -0.1700

Data Index: 922044 (C4 (Web Text))

Max Activating Token Index: 589

Text #8

Max Range: 8.3760. Min Range: -8.3760

Max Act: 7.7823. Min Act: -0.1700

Data Index: 290833 (C4 (Web Text))

Max Activating Token Index: 363

Text #9

Max Range: 8.3760. Min Range: -8.3760

Max Act: 7.7510. Min Act: -0.1700

Data Index: 1254567 (C4 (Web Text))

Max Activating Token Index: 733

Text #10

Max Range: 8.3760. Min Range: -8.3760

Max Act: 7.7823. Min Act: -0.1700

Data Index: 687020 (C4 (Web Text))

Max Activating Token Index: 324

Text #11

Max Range: 8.3760. Min Range: -8.3760

Max Act: 7.7198. Min Act: -0.1700

Data Index: 16129 (C4 (Web Text))

Max Activating Token Index: 964

Text #12

Max Range: 8.3760. Min Range: -8.3760

Max Act: 7.6885. Min Act: -0.1700

Data Index: 869846 (C4 (Web Text))

Max Activating Token Index: 215

Text #13

Max Range: 8.3760. Min Range: -8.3760

Max Act: 7.6260. Min Act: -0.1700

Data Index: 1014505 (C4 (Web Text))

Max Activating Token Index: 655

Text #14

Max Range: 8.3760. Min Range: -8.3760

Max Act: 7.5948. Min Act: -0.1700

Data Index: 1291423 (C4 (Web Text))

Max Activating Token Index: 762

Text #15

Max Range: 8.3760. Min Range: -8.3760

Max Act: 7.5323. Min Act: -0.1700

Data Index: 1250610 (C4 (Web Text))

Max Activating Token Index: 794

Text #16

Max Range: 8.3760. Min Range: -8.3760

Max Act: 7.5010. Min Act: -0.1700

Data Index: 532989 (C4 (Web Text))

Max Activating Token Index: 704

Text #17

Max Range: 8.3760. Min Range: -8.3760

Max Act: 7.5635. Min Act: -0.1700

Data Index: 1329199 (C4 (Web Text))

Max Activating Token Index: 960

Text #18

Max Range: 8.3760. Min Range: -8.3760

Max Act: 7.4073. Min Act: -0.1700

Data Index: 582414 (C4 (Web Text))

Max Activating Token Index: 997

Text #19

Max Range: 8.3760. Min Range: -8.3760

Max Act: 7.4385. Min Act: -0.1700

Data Index: 1225766 (C4 (Web Text))

Max Activating Token Index: 252