Model: GELU Model: 3 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1958 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('gelu-3l')

Text #0

Max Range: 10.2275. Min Range: -10.2275

Max Act: 10.2275. Min Act: -0.1700

Data Index: 1133805 (C4 (Web Text))

Max Activating Token Index: 490

Text #1

Max Range: 10.2275. Min Range: -10.2275

Max Act: 10.1025. Min Act: -0.1700

Data Index: 1005842 (C4 (Web Text))

Max Activating Token Index: 668

Text #2

Max Range: 10.2275. Min Range: -10.2275

Max Act: 8.9775. Min Act: -0.1700

Data Index: 264432 (C4 (Web Text))

Max Activating Token Index: 874

Text #3

Max Range: 10.2275. Min Range: -10.2275

Max Act: 8.8525. Min Act: -0.1700

Data Index: 1098219 (C4 (Web Text))

Max Activating Token Index: 177

Text #4

Max Range: 10.2275. Min Range: -10.2275

Max Act: 8.7900. Min Act: -0.1700

Data Index: 92208 (C4 (Web Text))

Max Activating Token Index: 390

Text #5

Max Range: 10.2275. Min Range: -10.2275

Max Act: 8.8525. Min Act: -0.1700

Data Index: 478417 (C4 (Web Text))

Max Activating Token Index: 837

Text #6

Max Range: 10.2275. Min Range: -10.2275

Max Act: 8.6650. Min Act: -0.1700

Data Index: 758943 (C4 (Web Text))

Max Activating Token Index: 683

Text #7

Max Range: 10.2275. Min Range: -10.2275

Max Act: 8.5400. Min Act: -0.1700

Data Index: 191258 (C4 (Web Text))

Max Activating Token Index: 909

Text #8

Max Range: 10.2275. Min Range: -10.2275

Max Act: 8.5400. Min Act: -0.1700

Data Index: 1242712 (C4 (Web Text))

Max Activating Token Index: 233

Text #9

Max Range: 10.2275. Min Range: -10.2275

Max Act: 8.5400. Min Act: -0.1696

Data Index: 1187499 (C4 (Web Text))

Max Activating Token Index: 106

Text #10

Max Range: 10.2275. Min Range: -10.2275

Max Act: 8.4463. Min Act: -0.1700

Data Index: 1272934 (C4 (Web Text))

Max Activating Token Index: 346

Text #11

Max Range: 10.2275. Min Range: -10.2275

Max Act: 8.6025. Min Act: -0.1700

Data Index: 555622 (C4 (Web Text))

Max Activating Token Index: 785

Text #12

Max Range: 10.2275. Min Range: -10.2275

Max Act: 8.6025. Min Act: -0.1700

Data Index: 38911 (C4 (Web Text))

Max Activating Token Index: 567

Text #13

Max Range: 10.2275. Min Range: -10.2275

Max Act: 8.4150. Min Act: -0.1700

Data Index: 1112478 (C4 (Web Text))

Max Activating Token Index: 110

Text #14

Max Range: 10.2275. Min Range: -10.2275

Max Act: 8.5713. Min Act: -0.1700

Data Index: 1336085 (C4 (Web Text))

Max Activating Token Index: 728

Text #15

Max Range: 10.2275. Min Range: -10.2275

Max Act: 8.4463. Min Act: -0.1700

Data Index: 1271985 (C4 (Web Text))

Max Activating Token Index: 155

Text #16

Max Range: 10.2275. Min Range: -10.2275

Max Act: 8.3838. Min Act: -0.1700

Data Index: 1302438 (C4 (Web Text))

Max Activating Token Index: 112

Text #17

Max Range: 10.2275. Min Range: -10.2275

Max Act: 8.4150. Min Act: -0.1700

Data Index: 22579 (C4 (Web Text))

Max Activating Token Index: 56

Text #18

Max Range: 10.2275. Min Range: -10.2275

Max Act: 8.5088. Min Act: -0.1700

Data Index: 972108 (C4 (Web Text))

Max Activating Token Index: 560

Text #19

Max Range: 10.2275. Min Range: -10.2275

Max Act: 8.4775. Min Act: -0.1700

Data Index: 227058 (C4 (Web Text))

Max Activating Token Index: 260