Model: GELU Model: 4 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1010 in Layer 3

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('gelu-4l')

Text #0

Max Range: 7.3757. Min Range: -7.3757

Max Act: 7.3757. Min Act: -0.1700

Data Index: 711525 (C4 (Web Text))

Max Activating Token Index: 548

Text #1

Max Range: 7.3757. Min Range: -7.3757

Max Act: 7.1569. Min Act: -0.1700

Data Index: 250618 (C4 (Web Text))

Max Activating Token Index: 730

Text #2

Max Range: 7.3757. Min Range: -7.3757

Max Act: 7.0632. Min Act: -0.1700

Data Index: 1087808 (C4 (Web Text))

Max Activating Token Index: 993

Text #3

Max Range: 7.3757. Min Range: -7.3757

Max Act: 6.9382. Min Act: -0.1700

Data Index: 93849 (C4 (Web Text))

Max Activating Token Index: 986

Text #4

Max Range: 7.3757. Min Range: -7.3757

Max Act: 6.8757. Min Act: -0.1700

Data Index: 635467 (C4 (Web Text))

Max Activating Token Index: 473

Text #5

Max Range: 7.3757. Min Range: -7.3757

Max Act: 6.9382. Min Act: -0.1700

Data Index: 800117 (C4 (Web Text))

Max Activating Token Index: 547

Text #6

Max Range: 7.3757. Min Range: -7.3757

Max Act: 6.8444. Min Act: -0.1700

Data Index: 390392 (C4 (Web Text))

Max Activating Token Index: 1003

Text #7

Max Range: 7.3757. Min Range: -7.3757

Max Act: 6.8444. Min Act: -0.1700

Data Index: 1280314 (C4 (Web Text))

Max Activating Token Index: 410

Text #8

Max Range: 7.3757. Min Range: -7.3757

Max Act: 6.9694. Min Act: -0.1700

Data Index: 31288 (C4 (Web Text))

Max Activating Token Index: 1017

Text #9

Max Range: 7.3757. Min Range: -7.3757

Max Act: 6.8132. Min Act: -0.1700

Data Index: 1287567 (C4 (Web Text))

Max Activating Token Index: 380

Text #10

Max Range: 7.3757. Min Range: -7.3757

Max Act: 6.7819. Min Act: -0.1700

Data Index: 1332914 (C4 (Web Text))

Max Activating Token Index: 758

Text #11

Max Range: 7.3757. Min Range: -7.3757

Max Act: 6.7507. Min Act: -0.1700

Data Index: 443825 (C4 (Web Text))

Max Activating Token Index: 881

Text #12

Max Range: 7.3757. Min Range: -7.3757

Max Act: 6.7507. Min Act: -0.1700

Data Index: 576582 (C4 (Web Text))

Max Activating Token Index: 498

Text #13

Max Range: 7.3757. Min Range: -7.3757

Max Act: 6.6882. Min Act: -0.1700

Data Index: 737336 (C4 (Web Text))

Max Activating Token Index: 710

Text #14

Max Range: 7.3757. Min Range: -7.3757

Max Act: 6.6569. Min Act: -0.1700

Data Index: 131812 (C4 (Web Text))

Max Activating Token Index: 351

Text #15

Max Range: 7.3757. Min Range: -7.3757

Max Act: 6.7194. Min Act: -0.1700

Data Index: 355916 (C4 (Web Text))

Max Activating Token Index: 881

Text #16

Max Range: 7.3757. Min Range: -7.3757

Max Act: 6.6569. Min Act: -0.1700

Data Index: 1115938 (C4 (Web Text))

Max Activating Token Index: 249

Text #17

Max Range: 7.3757. Min Range: -7.3757

Max Act: 6.7194. Min Act: -0.1700

Data Index: 262694 (C4 (Web Text))

Max Activating Token Index: 73

Text #18

Max Range: 7.3757. Min Range: -7.3757

Max Act: 6.6882. Min Act: -0.1700

Data Index: 270537 (C4 (Web Text))

Max Activating Token Index: 973

Text #19

Max Range: 7.3757. Min Range: -7.3757

Max Act: 6.5944. Min Act: -0.1700

Data Index: 691761 (C4 (Web Text))

Max Activating Token Index: 574