Model: GELU Model: 3 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 317 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('gelu-3l')

Text #0

Max Range: 7.6099. Min Range: -7.6099

Max Act: 7.6099. Min Act: -0.1700

Data Index: 236706 (C4 (Web Text))

Max Activating Token Index: 720

Text #1

Max Range: 7.6099. Min Range: -7.6099

Max Act: 7.4537. Min Act: -0.1700

Data Index: 1036680 (C4 (Web Text))

Max Activating Token Index: 448

Text #2

Max Range: 7.6099. Min Range: -7.6099

Max Act: 7.4224. Min Act: -0.1700

Data Index: 1335211 (C4 (Web Text))

Max Activating Token Index: 880

Text #3

Max Range: 7.6099. Min Range: -7.6099

Max Act: 7.4224. Min Act: -0.1700

Data Index: 1083658 (C4 (Web Text))

Max Activating Token Index: 595

Text #4

Max Range: 7.6099. Min Range: -7.6099

Max Act: 7.4224. Min Act: -0.1700

Data Index: 372653 (C4 (Web Text))

Max Activating Token Index: 923

Text #5

Max Range: 7.6099. Min Range: -7.6099

Max Act: 7.3287. Min Act: -0.1699

Data Index: 1263962 (C4 (Web Text))

Max Activating Token Index: 932

Text #6

Max Range: 7.6099. Min Range: -7.6099

Max Act: 7.3287. Min Act: -0.1700

Data Index: 523762 (C4 (Web Text))

Max Activating Token Index: 638

Text #7

Max Range: 7.6099. Min Range: -7.6099

Max Act: 7.2974. Min Act: -0.1696

Data Index: 1184502 (C4 (Web Text))

Max Activating Token Index: 445

Text #8

Max Range: 7.6099. Min Range: -7.6099

Max Act: 7.2974. Min Act: -0.1699

Data Index: 585719 (C4 (Web Text))

Max Activating Token Index: 135

Text #9

Max Range: 7.6099. Min Range: -7.6099

Max Act: 7.2349. Min Act: -0.1699

Data Index: 671914 (C4 (Web Text))

Max Activating Token Index: 415

Text #10

Max Range: 7.6099. Min Range: -7.6099

Max Act: 7.3287. Min Act: -0.1700

Data Index: 310436 (C4 (Web Text))

Max Activating Token Index: 877

Text #11

Max Range: 7.6099. Min Range: -7.6099

Max Act: 7.3287. Min Act: -0.1700

Data Index: 162386 (C4 (Web Text))

Max Activating Token Index: 470

Text #12

Max Range: 7.6099. Min Range: -7.6099

Max Act: 7.2349. Min Act: -0.1700

Data Index: 1002220 (C4 (Web Text))

Max Activating Token Index: 877

Text #13

Max Range: 7.6099. Min Range: -7.6099

Max Act: 7.2974. Min Act: -0.1700

Data Index: 932746 (C4 (Web Text))

Max Activating Token Index: 244

Text #14

Max Range: 7.6099. Min Range: -7.6099

Max Act: 7.2037. Min Act: -0.1700

Data Index: 113288 (C4 (Web Text))

Max Activating Token Index: 866

Text #15

Max Range: 7.6099. Min Range: -7.6099

Max Act: 7.2349. Min Act: -0.1694

Data Index: 452568 (C4 (Web Text))

Max Activating Token Index: 594

Text #16

Max Range: 7.6099. Min Range: -7.6099

Max Act: 7.2037. Min Act: -0.1700

Data Index: 1131259 (C4 (Web Text))

Max Activating Token Index: 409

Text #17

Max Range: 7.6099. Min Range: -7.6099

Max Act: 7.2037. Min Act: -0.1700

Data Index: 978385 (C4 (Web Text))

Max Activating Token Index: 648

Text #18

Max Range: 7.6099. Min Range: -7.6099

Max Act: 7.2349. Min Act: -0.1700

Data Index: 1260441 (C4 (Web Text))

Max Activating Token Index: 677

Text #19

Max Range: 7.6099. Min Range: -7.6099

Max Act: 7.2037. Min Act: -0.1700

Data Index: 805079 (C4 (Web Text))

Max Activating Token Index: 296