Model: GELU Model: 4 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2004 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('gelu-4l')

Text #0

Max Range: 3.3896. Min Range: -3.3896

Max Act: 3.3896. Min Act: -0.1700

Data Index: 898571 (C4 (Web Text))

Max Activating Token Index: 894

Text #1

Max Range: 3.3896. Min Range: -3.3896

Max Act: 3.3896. Min Act: -0.1700

Data Index: 598929 (C4 (Web Text))

Max Activating Token Index: 234

Text #2

Max Range: 3.3896. Min Range: -3.3896

Max Act: 3.3582. Min Act: -0.1700

Data Index: 341854 (C4 (Web Text))

Max Activating Token Index: 177

Text #3

Max Range: 3.3896. Min Range: -3.3896

Max Act: 3.3425. Min Act: -0.1700

Data Index: 231465 (C4 (Web Text))

Max Activating Token Index: 477

Text #4

Max Range: 3.3896. Min Range: -3.3896

Max Act: 3.3111. Min Act: -0.1700

Data Index: 183519 (C4 (Web Text))

Max Activating Token Index: 52

Text #5

Max Range: 3.3896. Min Range: -3.3896

Max Act: 3.3111. Min Act: -0.1700

Data Index: 944893 (C4 (Web Text))

Max Activating Token Index: 185

Text #6

Max Range: 3.3896. Min Range: -3.3896

Max Act: 3.2797. Min Act: -0.1700

Data Index: 1356399 (C4 (Web Text))

Max Activating Token Index: 449

Text #7

Max Range: 3.3896. Min Range: -3.3896

Max Act: 3.2797. Min Act: -0.1700

Data Index: 993369 (C4 (Web Text))

Max Activating Token Index: 213

Text #8

Max Range: 3.3896. Min Range: -3.3896

Max Act: 3.2640. Min Act: -0.1700

Data Index: 1243742 (C4 (Web Text))

Max Activating Token Index: 916

Text #9

Max Range: 3.3896. Min Range: -3.3896

Max Act: 3.2168. Min Act: -0.1700

Data Index: 155710 (C4 (Web Text))

Max Activating Token Index: 687

Text #10

Max Range: 3.3896. Min Range: -3.3896

Max Act: 3.2483. Min Act: -0.1700

Data Index: 171649 (C4 (Web Text))

Max Activating Token Index: 681

Text #11

Max Range: 3.3896. Min Range: -3.3896

Max Act: 3.2326. Min Act: -0.1700

Data Index: 426496 (C4 (Web Text))

Max Activating Token Index: 663

Text #12

Max Range: 3.3896. Min Range: -3.3896

Max Act: 3.2326. Min Act: -0.1700

Data Index: 1178699 (C4 (Web Text))

Max Activating Token Index: 567

Text #13

Max Range: 3.3896. Min Range: -3.3896

Max Act: 3.2011. Min Act: -0.1700

Data Index: 841573 (C4 (Web Text))

Max Activating Token Index: 189

Text #14

Max Range: 3.3896. Min Range: -3.3896

Max Act: 3.2011. Min Act: -0.1700

Data Index: 384367 (C4 (Web Text))

Max Activating Token Index: 734

Text #15

Max Range: 3.3896. Min Range: -3.3896

Max Act: 3.1696. Min Act: -0.1700

Data Index: 1305334 (C4 (Web Text))

Max Activating Token Index: 336

Text #16

Max Range: 3.3896. Min Range: -3.3896

Max Act: 3.1854. Min Act: -0.1700

Data Index: 1184026 (C4 (Web Text))

Max Activating Token Index: 71

Text #17

Max Range: 3.3896. Min Range: -3.3896

Max Act: 3.1854. Min Act: -0.1700

Data Index: 731485 (C4 (Web Text))

Max Activating Token Index: 510

Text #18

Max Range: 3.3896. Min Range: -3.3896

Max Act: 3.1696. Min Act: -0.1700

Data Index: 959400 (C4 (Web Text))

Max Activating Token Index: 98

Text #19

Max Range: 3.3896. Min Range: -3.3896

Max Act: 3.1696. Min Act: -0.1700

Data Index: 81832 (C4 (Web Text))

Max Activating Token Index: 304