Model: GELU Model: 3 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 358 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('gelu-3l')

Text #0

Max Range: 5.6709. Min Range: -5.6709

Max Act: 5.6709. Min Act: -0.1700

Data Index: 755318 (C4 (Web Text))

Max Activating Token Index: 145

Text #1

Max Range: 5.6709. Min Range: -5.6709

Max Act: 5.5772. Min Act: -0.1700

Data Index: 1202156 (C4 (Web Text))

Max Activating Token Index: 619

Text #2

Max Range: 5.6709. Min Range: -5.6709

Max Act: 5.4834. Min Act: -0.1700

Data Index: 514036 (C4 (Web Text))

Max Activating Token Index: 94

Text #3

Max Range: 5.6709. Min Range: -5.6709

Max Act: 5.4522. Min Act: -0.1700

Data Index: 177748 (C4 (Web Text))

Max Activating Token Index: 788

Text #4

Max Range: 5.6709. Min Range: -5.6709

Max Act: 5.3897. Min Act: -0.1700

Data Index: 305046 (C4 (Web Text))

Max Activating Token Index: 583

Text #5

Max Range: 5.6709. Min Range: -5.6709

Max Act: 5.4209. Min Act: -0.1700

Data Index: 1115233 (C4 (Web Text))

Max Activating Token Index: 708

Text #6

Max Range: 5.6709. Min Range: -5.6709

Max Act: 5.3897. Min Act: -0.1700

Data Index: 964770 (C4 (Web Text))

Max Activating Token Index: 802

Text #7

Max Range: 5.6709. Min Range: -5.6709

Max Act: 5.2959. Min Act: -0.1700

Data Index: 1482532 (Python Code)

Max Activating Token Index: 928

Text #8

Max Range: 5.6709. Min Range: -5.6709

Max Act: 5.3272. Min Act: -0.1700

Data Index: 803493 (C4 (Web Text))

Max Activating Token Index: 634

Text #9

Max Range: 5.6709. Min Range: -5.6709

Max Act: 5.2647. Min Act: -0.1700

Data Index: 393953 (C4 (Web Text))

Max Activating Token Index: 639

Text #10

Max Range: 5.6709. Min Range: -5.6709

Max Act: 5.2647. Min Act: -0.1700

Data Index: 790393 (C4 (Web Text))

Max Activating Token Index: 598

Text #11

Max Range: 5.6709. Min Range: -5.6709

Max Act: 5.2647. Min Act: -0.1700

Data Index: 1534353 (Python Code)

Max Activating Token Index: 999

Text #12

Max Range: 5.6709. Min Range: -5.6709

Max Act: 5.2022. Min Act: -0.1700

Data Index: 402035 (C4 (Web Text))

Max Activating Token Index: 382

Text #13

Max Range: 5.6709. Min Range: -5.6709

Max Act: 5.1709. Min Act: -0.1700

Data Index: 427191 (C4 (Web Text))

Max Activating Token Index: 452

Text #14

Max Range: 5.6709. Min Range: -5.6709

Max Act: 5.2022. Min Act: -0.1700

Data Index: 932850 (C4 (Web Text))

Max Activating Token Index: 326

Text #15

Max Range: 5.6709. Min Range: -5.6709

Max Act: 5.2022. Min Act: -0.1700

Data Index: 249257 (C4 (Web Text))

Max Activating Token Index: 575

Text #16

Max Range: 5.6709. Min Range: -5.6709

Max Act: 5.2022. Min Act: -0.1700

Data Index: 1625519 (Python Code)

Max Activating Token Index: 973

Text #17

Max Range: 5.6709. Min Range: -5.6709

Max Act: 5.1709. Min Act: -0.1700

Data Index: 698601 (C4 (Web Text))

Max Activating Token Index: 620

Text #18

Max Range: 5.6709. Min Range: -5.6709

Max Act: 5.1397. Min Act: -0.1700

Data Index: 1400709 (Python Code)

Max Activating Token Index: 514

Text #19

Max Range: 5.6709. Min Range: -5.6709

Max Act: 5.1397. Min Act: -0.1700

Data Index: 1571124 (Python Code)

Max Activating Token Index: 619