Model: GELU Model: 4 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1755 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('gelu-4l')

Text #0

Max Range: 5.5978. Min Range: -5.5978

Max Act: 5.5978. Min Act: -0.1700

Data Index: 415520 (C4 (Web Text))

Max Activating Token Index: 625

Text #1

Max Range: 5.5978. Min Range: -5.5978

Max Act: 5.5041. Min Act: -0.1700

Data Index: 579630 (C4 (Web Text))

Max Activating Token Index: 590

Text #2

Max Range: 5.5978. Min Range: -5.5978

Max Act: 5.4728. Min Act: -0.1700

Data Index: 1053759 (C4 (Web Text))

Max Activating Token Index: 825

Text #3

Max Range: 5.5978. Min Range: -5.5978

Max Act: 5.3791. Min Act: -0.1700

Data Index: 1154252 (C4 (Web Text))

Max Activating Token Index: 992

Text #4

Max Range: 5.5978. Min Range: -5.5978

Max Act: 5.3478. Min Act: -0.1700

Data Index: 626919 (C4 (Web Text))

Max Activating Token Index: 572

Text #5

Max Range: 5.5978. Min Range: -5.5978

Max Act: 5.2853. Min Act: -0.1700

Data Index: 1203463 (C4 (Web Text))

Max Activating Token Index: 201

Text #6

Max Range: 5.5978. Min Range: -5.5978

Max Act: 5.2541. Min Act: -0.1700

Data Index: 505045 (C4 (Web Text))

Max Activating Token Index: 432

Text #7

Max Range: 5.5978. Min Range: -5.5978

Max Act: 5.2541. Min Act: -0.1700

Data Index: 1012616 (C4 (Web Text))

Max Activating Token Index: 738

Text #8

Max Range: 5.5978. Min Range: -5.5978

Max Act: 5.2541. Min Act: -0.1700

Data Index: 705062 (C4 (Web Text))

Max Activating Token Index: 779

Text #9

Max Range: 5.5978. Min Range: -5.5978

Max Act: 5.2228. Min Act: -0.1700

Data Index: 118545 (C4 (Web Text))

Max Activating Token Index: 584

Text #10

Max Range: 5.5978. Min Range: -5.5978

Max Act: 5.2541. Min Act: -0.1700

Data Index: 325794 (C4 (Web Text))

Max Activating Token Index: 579

Text #11

Max Range: 5.5978. Min Range: -5.5978

Max Act: 5.2541. Min Act: -0.1700

Data Index: 1166173 (C4 (Web Text))

Max Activating Token Index: 349

Text #12

Max Range: 5.5978. Min Range: -5.5978

Max Act: 5.2228. Min Act: -0.1700

Data Index: 413701 (C4 (Web Text))

Max Activating Token Index: 738

Text #13

Max Range: 5.5978. Min Range: -5.5978

Max Act: 5.2228. Min Act: -0.1700

Data Index: 91177 (C4 (Web Text))

Max Activating Token Index: 228

Text #14

Max Range: 5.5978. Min Range: -5.5978

Max Act: 5.2541. Min Act: -0.1700

Data Index: 370826 (C4 (Web Text))

Max Activating Token Index: 592

Text #15

Max Range: 5.5978. Min Range: -5.5978

Max Act: 5.2541. Min Act: -0.1700

Data Index: 733771 (C4 (Web Text))

Max Activating Token Index: 601

Text #16

Max Range: 5.5978. Min Range: -5.5978

Max Act: 5.1916. Min Act: -0.1700

Data Index: 311481 (C4 (Web Text))

Max Activating Token Index: 947

Text #17

Max Range: 5.5978. Min Range: -5.5978

Max Act: 5.2541. Min Act: -0.1700

Data Index: 472847 (C4 (Web Text))

Max Activating Token Index: 470

Text #18

Max Range: 5.5978. Min Range: -5.5978

Max Act: 5.1916. Min Act: -0.1700

Data Index: 53998 (C4 (Web Text))

Max Activating Token Index: 302

Text #19

Max Range: 5.5978. Min Range: -5.5978

Max Act: 5.1916. Min Act: -0.1700

Data Index: 836170 (C4 (Web Text))

Max Activating Token Index: 581