Model: GELU Model: 4 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 271 in Layer 1

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('gelu-4l')

Text #0

Max Range: 5.5998. Min Range: -5.5998

Max Act: 5.5998. Min Act: -0.1700

Data Index: 1101108 (C4 (Web Text))

Max Activating Token Index: 213

Text #1

Max Range: 5.5998. Min Range: -5.5998

Max Act: 5.5060. Min Act: -0.1700

Data Index: 474426 (C4 (Web Text))

Max Activating Token Index: 644

Text #2

Max Range: 5.5998. Min Range: -5.5998

Max Act: 5.5060. Min Act: -0.1700

Data Index: 561142 (C4 (Web Text))

Max Activating Token Index: 872

Text #3

Max Range: 5.5998. Min Range: -5.5998

Max Act: 5.5373. Min Act: -0.1700

Data Index: 101112 (C4 (Web Text))

Max Activating Token Index: 797

Text #4

Max Range: 5.5998. Min Range: -5.5998

Max Act: 5.4748. Min Act: -0.1700

Data Index: 428667 (C4 (Web Text))

Max Activating Token Index: 833

Text #5

Max Range: 5.5998. Min Range: -5.5998

Max Act: 5.4435. Min Act: -0.1700

Data Index: 397994 (C4 (Web Text))

Max Activating Token Index: 665

Text #6

Max Range: 5.5998. Min Range: -5.5998

Max Act: 5.4748. Min Act: -0.1700

Data Index: 812046 (C4 (Web Text))

Max Activating Token Index: 672

Text #7

Max Range: 5.5998. Min Range: -5.5998

Max Act: 5.4435. Min Act: -0.1700

Data Index: 128271 (C4 (Web Text))

Max Activating Token Index: 493

Text #8

Max Range: 5.5998. Min Range: -5.5998

Max Act: 5.4435. Min Act: -0.1700

Data Index: 1253663 (C4 (Web Text))

Max Activating Token Index: 777

Text #9

Max Range: 5.5998. Min Range: -5.5998

Max Act: 5.4748. Min Act: -0.1700

Data Index: 276048 (C4 (Web Text))

Max Activating Token Index: 206

Text #10

Max Range: 5.5998. Min Range: -5.5998

Max Act: 5.3810. Min Act: -0.1700

Data Index: 632203 (C4 (Web Text))

Max Activating Token Index: 294

Text #11

Max Range: 5.5998. Min Range: -5.5998

Max Act: 5.3810. Min Act: -0.1700

Data Index: 200931 (C4 (Web Text))

Max Activating Token Index: 598

Text #12

Max Range: 5.5998. Min Range: -5.5998

Max Act: 5.3810. Min Act: -0.1700

Data Index: 111875 (C4 (Web Text))

Max Activating Token Index: 123

Text #13

Max Range: 5.5998. Min Range: -5.5998

Max Act: 5.3498. Min Act: -0.1700

Data Index: 1357536 (C4 (Web Text))

Max Activating Token Index: 445

Text #14

Max Range: 5.5998. Min Range: -5.5998

Max Act: 5.3498. Min Act: -0.1700

Data Index: 94472 (C4 (Web Text))

Max Activating Token Index: 203

Text #15

Max Range: 5.5998. Min Range: -5.5998

Max Act: 5.3185. Min Act: -0.1700

Data Index: 785433 (C4 (Web Text))

Max Activating Token Index: 463

Text #16

Max Range: 5.5998. Min Range: -5.5998

Max Act: 5.3498. Min Act: -0.1700

Data Index: 431158 (C4 (Web Text))

Max Activating Token Index: 702

Text #17

Max Range: 5.5998. Min Range: -5.5998

Max Act: 5.3185. Min Act: -0.1700

Data Index: 91333 (C4 (Web Text))

Max Activating Token Index: 858

Text #18

Max Range: 5.5998. Min Range: -5.5998

Max Act: 5.2873. Min Act: -0.1700

Data Index: 1161184 (C4 (Web Text))

Max Activating Token Index: 496

Text #19

Max Range: 5.5998. Min Range: -5.5998

Max Act: 5.3185. Min Act: -0.1700

Data Index: 839383 (C4 (Web Text))

Max Activating Token Index: 478