Model: GELU Model: 3 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1305 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('gelu-3l')

Text #0

Max Range: 5.8712. Min Range: -5.8712

Max Act: 5.8712. Min Act: -0.1700

Data Index: 245812 (C4 (Web Text))

Max Activating Token Index: 945

Text #1

Max Range: 5.8712. Min Range: -5.8712

Max Act: 5.7150. Min Act: -0.1699

Data Index: 778299 (C4 (Web Text))

Max Activating Token Index: 1001

Text #2

Max Range: 5.8712. Min Range: -5.8712

Max Act: 5.5275. Min Act: -0.1700

Data Index: 342767 (C4 (Web Text))

Max Activating Token Index: 592

Text #3

Max Range: 5.8712. Min Range: -5.8712

Max Act: 5.4650. Min Act: -0.1700

Data Index: 944158 (C4 (Web Text))

Max Activating Token Index: 718

Text #4

Max Range: 5.8712. Min Range: -5.8712

Max Act: 5.4337. Min Act: -0.1700

Data Index: 91059 (C4 (Web Text))

Max Activating Token Index: 198

Text #5

Max Range: 5.8712. Min Range: -5.8712

Max Act: 5.3712. Min Act: -0.1700

Data Index: 1045153 (C4 (Web Text))

Max Activating Token Index: 726

Text #6

Max Range: 5.8712. Min Range: -5.8712

Max Act: 5.2775. Min Act: -0.1699

Data Index: 722839 (C4 (Web Text))

Max Activating Token Index: 447

Text #7

Max Range: 5.8712. Min Range: -5.8712

Max Act: 5.2462. Min Act: -0.1700

Data Index: 28553 (C4 (Web Text))

Max Activating Token Index: 879

Text #8

Max Range: 5.8712. Min Range: -5.8712

Max Act: 5.2150. Min Act: -0.1700

Data Index: 719319 (C4 (Web Text))

Max Activating Token Index: 821

Text #9

Max Range: 5.8712. Min Range: -5.8712

Max Act: 5.2462. Min Act: -0.1700

Data Index: 265930 (C4 (Web Text))

Max Activating Token Index: 866

Text #10

Max Range: 5.8712. Min Range: -5.8712

Max Act: 5.2150. Min Act: -0.1700

Data Index: 102504 (C4 (Web Text))

Max Activating Token Index: 714

Text #11

Max Range: 5.8712. Min Range: -5.8712

Max Act: 5.2150. Min Act: -0.1700

Data Index: 419879 (C4 (Web Text))

Max Activating Token Index: 628

Text #12

Max Range: 5.8712. Min Range: -5.8712

Max Act: 5.2150. Min Act: -0.1700

Data Index: 335884 (C4 (Web Text))

Max Activating Token Index: 481

Text #13

Max Range: 5.8712. Min Range: -5.8712

Max Act: 5.1837. Min Act: -0.1700

Data Index: 175905 (C4 (Web Text))

Max Activating Token Index: 360

Text #14

Max Range: 5.8712. Min Range: -5.8712

Max Act: 5.2462. Min Act: -0.1699

Data Index: 415653 (C4 (Web Text))

Max Activating Token Index: 48

Text #15

Max Range: 5.8712. Min Range: -5.8712

Max Act: 5.1837. Min Act: -0.1700

Data Index: 835753 (C4 (Web Text))

Max Activating Token Index: 585

Text #16

Max Range: 5.8712. Min Range: -5.8712

Max Act: 5.1837. Min Act: -0.1700

Data Index: 378696 (C4 (Web Text))

Max Activating Token Index: 275

Text #17

Max Range: 5.8712. Min Range: -5.8712

Max Act: 5.1212. Min Act: -0.1700

Data Index: 845314 (C4 (Web Text))

Max Activating Token Index: 70

Text #18

Max Range: 5.8712. Min Range: -5.8712

Max Act: 5.1525. Min Act: -0.1698

Data Index: 891155 (C4 (Web Text))

Max Activating Token Index: 116

Text #19

Max Range: 5.8712. Min Range: -5.8712

Max Act: 5.0900. Min Act: -0.1700

Data Index: 375768 (C4 (Web Text))

Max Activating Token Index: 257