Model: GELU Model: 4 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1239 in Layer 3

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('gelu-4l')

Text #0

Max Range: 5.2812. Min Range: -5.2812

Max Act: 5.2812. Min Act: -0.1700

Data Index: 885507 (C4 (Web Text))

Max Activating Token Index: 911

Text #1

Max Range: 5.2812. Min Range: -5.2812

Max Act: 5.2812. Min Act: -0.1700

Data Index: 446353 (C4 (Web Text))

Max Activating Token Index: 1021

Text #2

Max Range: 5.2812. Min Range: -5.2812

Max Act: 5.2499. Min Act: -0.1700

Data Index: 6191 (C4 (Web Text))

Max Activating Token Index: 63

Text #3

Max Range: 5.2812. Min Range: -5.2812

Max Act: 5.1562. Min Act: -0.1700

Data Index: 620714 (C4 (Web Text))

Max Activating Token Index: 163

Text #4

Max Range: 5.2812. Min Range: -5.2812

Max Act: 5.0780. Min Act: -0.1700

Data Index: 699664 (C4 (Web Text))

Max Activating Token Index: 21

Text #5

Max Range: 5.2812. Min Range: -5.2812

Max Act: 5.0937. Min Act: -0.1700

Data Index: 1603318 (Python Code)

Max Activating Token Index: 42

Text #6

Max Range: 5.2812. Min Range: -5.2812

Max Act: 5.0624. Min Act: -0.1700

Data Index: 1232310 (C4 (Web Text))

Max Activating Token Index: 603

Text #7

Max Range: 5.2812. Min Range: -5.2812

Max Act: 4.9843. Min Act: -0.1700

Data Index: 1133152 (C4 (Web Text))

Max Activating Token Index: 482

Text #8

Max Range: 5.2812. Min Range: -5.2812

Max Act: 4.9999. Min Act: -0.1700

Data Index: 1328644 (C4 (Web Text))

Max Activating Token Index: 134

Text #9

Max Range: 5.2812. Min Range: -5.2812

Max Act: 4.9374. Min Act: -0.1700

Data Index: 509586 (C4 (Web Text))

Max Activating Token Index: 273

Text #10

Max Range: 5.2812. Min Range: -5.2812

Max Act: 4.9530. Min Act: -0.1700

Data Index: 550933 (C4 (Web Text))

Max Activating Token Index: 23

Text #11

Max Range: 5.2812. Min Range: -5.2812

Max Act: 4.8593. Min Act: -0.1700

Data Index: 946749 (C4 (Web Text))

Max Activating Token Index: 258

Text #12

Max Range: 5.2812. Min Range: -5.2812

Max Act: 4.8905. Min Act: -0.1700

Data Index: 455025 (C4 (Web Text))

Max Activating Token Index: 526

Text #13

Max Range: 5.2812. Min Range: -5.2812

Max Act: 4.8905. Min Act: -0.1700

Data Index: 1298113 (C4 (Web Text))

Max Activating Token Index: 322

Text #14

Max Range: 5.2812. Min Range: -5.2812

Max Act: 4.9218. Min Act: -0.1700

Data Index: 700019 (C4 (Web Text))

Max Activating Token Index: 874

Text #15

Max Range: 5.2812. Min Range: -5.2812

Max Act: 4.8749. Min Act: -0.1700

Data Index: 455641 (C4 (Web Text))

Max Activating Token Index: 407

Text #16

Max Range: 5.2812. Min Range: -5.2812

Max Act: 4.8437. Min Act: -0.1700

Data Index: 987992 (C4 (Web Text))

Max Activating Token Index: 52

Text #17

Max Range: 5.2812. Min Range: -5.2812

Max Act: 4.7968. Min Act: -0.1700

Data Index: 974390 (C4 (Web Text))

Max Activating Token Index: 1002

Text #18

Max Range: 5.2812. Min Range: -5.2812

Max Act: 4.7968. Min Act: -0.1700

Data Index: 730214 (C4 (Web Text))

Max Activating Token Index: 376

Text #19

Max Range: 5.2812. Min Range: -5.2812

Max Act: 4.8593. Min Act: -0.1700

Data Index: 761047 (C4 (Web Text))

Max Activating Token Index: 490