Model: GELU Model: 4 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1641 in Layer 0

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('gelu-4l')

Text #0

Max Range: 3.4290. Min Range: -3.4290

Max Act: 3.4290. Min Act: -0.1700

Data Index: 1150814 (C4 (Web Text))

Max Activating Token Index: 712

Text #1

Max Range: 3.4290. Min Range: -3.4290

Max Act: 3.3191. Min Act: -0.1700

Data Index: 775290 (C4 (Web Text))

Max Activating Token Index: 213

Text #2

Max Range: 3.4290. Min Range: -3.4290

Max Act: 3.2406. Min Act: -0.1700

Data Index: 1175226 (C4 (Web Text))

Max Activating Token Index: 189

Text #3

Max Range: 3.4290. Min Range: -3.4290

Max Act: 3.2563. Min Act: -0.1700

Data Index: 1032286 (C4 (Web Text))

Max Activating Token Index: 264

Text #4

Max Range: 3.4290. Min Range: -3.4290

Max Act: 3.2249. Min Act: -0.1700

Data Index: 100952 (C4 (Web Text))

Max Activating Token Index: 185

Text #5

Max Range: 3.4290. Min Range: -3.4290

Max Act: 3.2249. Min Act: -0.1700

Data Index: 119706 (C4 (Web Text))

Max Activating Token Index: 781

Text #6

Max Range: 3.4290. Min Range: -3.4290

Max Act: 3.2406. Min Act: -0.1700

Data Index: 1352873 (C4 (Web Text))

Max Activating Token Index: 927

Text #7

Max Range: 3.4290. Min Range: -3.4290

Max Act: 3.2406. Min Act: -0.1700

Data Index: 1269613 (C4 (Web Text))

Max Activating Token Index: 165

Text #8

Max Range: 3.4290. Min Range: -3.4290

Max Act: 3.2091. Min Act: -0.1700

Data Index: 410375 (C4 (Web Text))

Max Activating Token Index: 783

Text #9

Max Range: 3.4290. Min Range: -3.4290

Max Act: 3.2249. Min Act: -0.1700

Data Index: 595233 (C4 (Web Text))

Max Activating Token Index: 268

Text #10

Max Range: 3.4290. Min Range: -3.4290

Max Act: 3.2249. Min Act: -0.1700

Data Index: 645443 (C4 (Web Text))

Max Activating Token Index: 213

Text #11

Max Range: 3.4290. Min Range: -3.4290

Max Act: 3.2406. Min Act: -0.1700

Data Index: 401558 (C4 (Web Text))

Max Activating Token Index: 415

Text #12

Max Range: 3.4290. Min Range: -3.4290

Max Act: 3.2406. Min Act: -0.1700

Data Index: 56746 (C4 (Web Text))

Max Activating Token Index: 1002

Text #13

Max Range: 3.4290. Min Range: -3.4290

Max Act: 3.2406. Min Act: -0.1700

Data Index: 1208089 (C4 (Web Text))

Max Activating Token Index: 5

Text #14

Max Range: 3.4290. Min Range: -3.4290

Max Act: 3.2249. Min Act: -0.1700

Data Index: 115194 (C4 (Web Text))

Max Activating Token Index: 164

Text #15

Max Range: 3.4290. Min Range: -3.4290

Max Act: 3.2406. Min Act: -0.1700

Data Index: 111032 (C4 (Web Text))

Max Activating Token Index: 80

Text #16

Max Range: 3.4290. Min Range: -3.4290

Max Act: 3.2406. Min Act: -0.1700

Data Index: 1068439 (C4 (Web Text))

Max Activating Token Index: 254

Text #17

Max Range: 3.4290. Min Range: -3.4290

Max Act: 3.2249. Min Act: -0.1700

Data Index: 521648 (C4 (Web Text))

Max Activating Token Index: 421

Text #18

Max Range: 3.4290. Min Range: -3.4290

Max Act: 3.2249. Min Act: -0.1700

Data Index: 366633 (C4 (Web Text))

Max Activating Token Index: 464

Text #19

Max Range: 3.4290. Min Range: -3.4290

Max Act: 3.2249. Min Act: -0.1700

Data Index: 120763 (C4 (Web Text))

Max Activating Token Index: 703