Model: GELU Model: 4 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2006 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('gelu-4l')

Text #0

Max Range: 5.1829. Min Range: -5.1829

Max Act: 5.1829. Min Act: -0.1699

Data Index: 1040997 (C4 (Web Text))

Max Activating Token Index: 887

Text #1

Max Range: 5.1829. Min Range: -5.1829

Max Act: 5.0266. Min Act: -0.1700

Data Index: 841179 (C4 (Web Text))

Max Activating Token Index: 739

Text #2

Max Range: 5.1829. Min Range: -5.1829

Max Act: 5.0266. Min Act: -0.1699

Data Index: 637345 (C4 (Web Text))

Max Activating Token Index: 777

Text #3

Max Range: 5.1829. Min Range: -5.1829

Max Act: 4.6829. Min Act: -0.1699

Data Index: 518711 (C4 (Web Text))

Max Activating Token Index: 999

Text #4

Max Range: 5.1829. Min Range: -5.1829

Max Act: 4.8079. Min Act: -0.1699

Data Index: 205468 (C4 (Web Text))

Max Activating Token Index: 1023

Text #5

Max Range: 5.1829. Min Range: -5.1829

Max Act: 4.6829. Min Act: -0.1700

Data Index: 377540 (C4 (Web Text))

Max Activating Token Index: 139

Text #6

Max Range: 5.1829. Min Range: -5.1829

Max Act: 4.6204. Min Act: -0.1699

Data Index: 42357 (C4 (Web Text))

Max Activating Token Index: 505

Text #7

Max Range: 5.1829. Min Range: -5.1829

Max Act: 4.6516. Min Act: -0.1699

Data Index: 802656 (C4 (Web Text))

Max Activating Token Index: 371

Text #8

Max Range: 5.1829. Min Range: -5.1829

Max Act: 4.8704. Min Act: -0.1699

Data Index: 935911 (C4 (Web Text))

Max Activating Token Index: 1023

Text #9

Max Range: 5.1829. Min Range: -5.1829

Max Act: 4.5266. Min Act: -0.1698

Data Index: 1318184 (C4 (Web Text))

Max Activating Token Index: 1023

Text #10

Max Range: 5.1829. Min Range: -5.1829

Max Act: 4.5891. Min Act: -0.1700

Data Index: 225256 (C4 (Web Text))

Max Activating Token Index: 582

Text #11

Max Range: 5.1829. Min Range: -5.1829

Max Act: 4.4954. Min Act: -0.1698

Data Index: 1142709 (C4 (Web Text))

Max Activating Token Index: 387

Text #12

Max Range: 5.1829. Min Range: -5.1829

Max Act: 4.5579. Min Act: -0.1699

Data Index: 548718 (C4 (Web Text))

Max Activating Token Index: 584

Text #13

Max Range: 5.1829. Min Range: -5.1829

Max Act: 4.5579. Min Act: -0.1698

Data Index: 940942 (C4 (Web Text))

Max Activating Token Index: 436

Text #14

Max Range: 5.1829. Min Range: -5.1829

Max Act: 4.5891. Min Act: -0.1700

Data Index: 116808 (C4 (Web Text))

Max Activating Token Index: 1023

Text #15

Max Range: 5.1829. Min Range: -5.1829

Max Act: 4.5579. Min Act: -0.1700

Data Index: 412617 (C4 (Web Text))

Max Activating Token Index: 365

Text #16

Max Range: 5.1829. Min Range: -5.1829

Max Act: 4.5266. Min Act: -0.1699

Data Index: 69600 (C4 (Web Text))

Max Activating Token Index: 1023

Text #17

Max Range: 5.1829. Min Range: -5.1829

Max Act: 4.5891. Min Act: -0.1699

Data Index: 453810 (C4 (Web Text))

Max Activating Token Index: 729

Text #18

Max Range: 5.1829. Min Range: -5.1829

Max Act: 4.5266. Min Act: -0.1699

Data Index: 208528 (C4 (Web Text))

Max Activating Token Index: 112

Text #19

Max Range: 5.1829. Min Range: -5.1829

Max Act: 4.5266. Min Act: -0.1699

Data Index: 326929 (C4 (Web Text))

Max Activating Token Index: 290