Model: GELU Model: 3 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1231 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('gelu-3l')

Text #0

Max Range: 6.3111. Min Range: -6.3111

Max Act: 6.3111. Min Act: -0.1700

Data Index: 355058 (C4 (Web Text))

Max Activating Token Index: 449

Text #1

Max Range: 6.3111. Min Range: -6.3111

Max Act: 6.2798. Min Act: -0.1700

Data Index: 528953 (C4 (Web Text))

Max Activating Token Index: 153

Text #2

Max Range: 6.3111. Min Range: -6.3111

Max Act: 6.2486. Min Act: -0.1700

Data Index: 759796 (C4 (Web Text))

Max Activating Token Index: 497

Text #3

Max Range: 6.3111. Min Range: -6.3111

Max Act: 6.2486. Min Act: -0.1700

Data Index: 1087078 (C4 (Web Text))

Max Activating Token Index: 974

Text #4

Max Range: 6.3111. Min Range: -6.3111

Max Act: 6.2486. Min Act: -0.1700

Data Index: 533770 (C4 (Web Text))

Max Activating Token Index: 903

Text #5

Max Range: 6.3111. Min Range: -6.3111

Max Act: 6.1861. Min Act: -0.1700

Data Index: 1008661 (C4 (Web Text))

Max Activating Token Index: 671

Text #6

Max Range: 6.3111. Min Range: -6.3111

Max Act: 6.2173. Min Act: -0.1700

Data Index: 492278 (C4 (Web Text))

Max Activating Token Index: 58

Text #7

Max Range: 6.3111. Min Range: -6.3111

Max Act: 6.1861. Min Act: -0.1700

Data Index: 983477 (C4 (Web Text))

Max Activating Token Index: 584

Text #8

Max Range: 6.3111. Min Range: -6.3111

Max Act: 6.1548. Min Act: -0.1700

Data Index: 1319725 (C4 (Web Text))

Max Activating Token Index: 146

Text #9

Max Range: 6.3111. Min Range: -6.3111

Max Act: 6.0923. Min Act: -0.1700

Data Index: 609866 (C4 (Web Text))

Max Activating Token Index: 705

Text #10

Max Range: 6.3111. Min Range: -6.3111

Max Act: 6.1236. Min Act: -0.1700

Data Index: 504949 (C4 (Web Text))

Max Activating Token Index: 196

Text #11

Max Range: 6.3111. Min Range: -6.3111

Max Act: 6.0611. Min Act: -0.1700

Data Index: 457738 (C4 (Web Text))

Max Activating Token Index: 628

Text #12

Max Range: 6.3111. Min Range: -6.3111

Max Act: 6.1548. Min Act: -0.1700

Data Index: 576199 (C4 (Web Text))

Max Activating Token Index: 848

Text #13

Max Range: 6.3111. Min Range: -6.3111

Max Act: 6.0611. Min Act: -0.1700

Data Index: 904294 (C4 (Web Text))

Max Activating Token Index: 923

Text #14

Max Range: 6.3111. Min Range: -6.3111

Max Act: 6.0611. Min Act: -0.1700

Data Index: 187040 (C4 (Web Text))

Max Activating Token Index: 131

Text #15

Max Range: 6.3111. Min Range: -6.3111

Max Act: 6.0298. Min Act: -0.1700

Data Index: 950962 (C4 (Web Text))

Max Activating Token Index: 348

Text #16

Max Range: 6.3111. Min Range: -6.3111

Max Act: 6.0611. Min Act: -0.1700

Data Index: 99388 (C4 (Web Text))

Max Activating Token Index: 40

Text #17

Max Range: 6.3111. Min Range: -6.3111

Max Act: 6.0611. Min Act: -0.1700

Data Index: 946930 (C4 (Web Text))

Max Activating Token Index: 508

Text #18

Max Range: 6.3111. Min Range: -6.3111

Max Act: 5.9986. Min Act: -0.1700

Data Index: 765474 (C4 (Web Text))

Max Activating Token Index: 251

Text #19

Max Range: 6.3111. Min Range: -6.3111

Max Act: 5.9673. Min Act: -0.1700

Data Index: 968648 (C4 (Web Text))

Max Activating Token Index: 440