Model: GELU Model: 2 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1744 in Layer 1

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('gelu-2l')

Text #0

Max Range: 4.0848. Min Range: -4.0848

Max Act: 4.0848. Min Act: -0.1700

Data Index: 498418 (C4 (Web Text))

Max Activating Token Index: 295

Text #1

Max Range: 4.0848. Min Range: -4.0848

Max Act: 4.0379. Min Act: -0.1700

Data Index: 956174 (C4 (Web Text))

Max Activating Token Index: 710

Text #2

Max Range: 4.0848. Min Range: -4.0848

Max Act: 4.0223. Min Act: -0.1700

Data Index: 200577 (C4 (Web Text))

Max Activating Token Index: 803

Text #3

Max Range: 4.0848. Min Range: -4.0848

Max Act: 3.9441. Min Act: -0.1700

Data Index: 1168225 (C4 (Web Text))

Max Activating Token Index: 50

Text #4

Max Range: 4.0848. Min Range: -4.0848

Max Act: 3.9128. Min Act: -0.1700

Data Index: 455999 (C4 (Web Text))

Max Activating Token Index: 671

Text #5

Max Range: 4.0848. Min Range: -4.0848

Max Act: 3.9128. Min Act: -0.1700

Data Index: 798634 (C4 (Web Text))

Max Activating Token Index: 502

Text #6

Max Range: 4.0848. Min Range: -4.0848

Max Act: 3.8816. Min Act: -0.1700

Data Index: 927091 (C4 (Web Text))

Max Activating Token Index: 34

Text #7

Max Range: 4.0848. Min Range: -4.0848

Max Act: 3.8503. Min Act: -0.1700

Data Index: 378469 (C4 (Web Text))

Max Activating Token Index: 451

Text #8

Max Range: 4.0848. Min Range: -4.0848

Max Act: 3.8190. Min Act: -0.1700

Data Index: 562114 (C4 (Web Text))

Max Activating Token Index: 88

Text #9

Max Range: 4.0848. Min Range: -4.0848

Max Act: 3.8347. Min Act: -0.1700

Data Index: 311903 (C4 (Web Text))

Max Activating Token Index: 42

Text #10

Max Range: 4.0848. Min Range: -4.0848

Max Act: 3.7565. Min Act: -0.1700

Data Index: 538289 (C4 (Web Text))

Max Activating Token Index: 690

Text #11

Max Range: 4.0848. Min Range: -4.0848

Max Act: 3.7721. Min Act: -0.1700

Data Index: 1001535 (C4 (Web Text))

Max Activating Token Index: 104

Text #12

Max Range: 4.0848. Min Range: -4.0848

Max Act: 3.8034. Min Act: -0.1700

Data Index: 965831 (C4 (Web Text))

Max Activating Token Index: 820

Text #13

Max Range: 4.0848. Min Range: -4.0848

Max Act: 3.8190. Min Act: -0.1700

Data Index: 1117768 (C4 (Web Text))

Max Activating Token Index: 970

Text #14

Max Range: 4.0848. Min Range: -4.0848

Max Act: 3.7408. Min Act: -0.1700

Data Index: 546309 (C4 (Web Text))

Max Activating Token Index: 146

Text #15

Max Range: 4.0848. Min Range: -4.0848

Max Act: 3.7408. Min Act: -0.1700

Data Index: 1170119 (C4 (Web Text))

Max Activating Token Index: 667

Text #16

Max Range: 4.0848. Min Range: -4.0848

Max Act: 3.7095. Min Act: -0.1700

Data Index: 221842 (C4 (Web Text))

Max Activating Token Index: 808

Text #17

Max Range: 4.0848. Min Range: -4.0848

Max Act: 3.7721. Min Act: -0.1700

Data Index: 708224 (C4 (Web Text))

Max Activating Token Index: 611

Text #18

Max Range: 4.0848. Min Range: -4.0848

Max Act: 3.7408. Min Act: -0.1700

Data Index: 598785 (C4 (Web Text))

Max Activating Token Index: 894

Text #19

Max Range: 4.0848. Min Range: -4.0848

Max Act: 3.6626. Min Act: -0.1700

Data Index: 33234 (C4 (Web Text))

Max Activating Token Index: 737