Model: GELU Model: 3 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1976 in Layer 0

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('gelu-3l')

Text #0

Max Range: 3.8902. Min Range: -3.8902

Max Act: 3.8902. Min Act: -0.1700

Data Index: 760699 (C4 (Web Text))

Max Activating Token Index: 895

Text #1

Max Range: 3.8902. Min Range: -3.8902

Max Act: 3.8589. Min Act: -0.1700

Data Index: 1299958 (C4 (Web Text))

Max Activating Token Index: 350

Text #2

Max Range: 3.8902. Min Range: -3.8902

Max Act: 3.8276. Min Act: -0.1700

Data Index: 965903 (C4 (Web Text))

Max Activating Token Index: 819

Text #3

Max Range: 3.8902. Min Range: -3.8902

Max Act: 3.7964. Min Act: -0.1700

Data Index: 164401 (C4 (Web Text))

Max Activating Token Index: 448

Text #4

Max Range: 3.8902. Min Range: -3.8902

Max Act: 3.7964. Min Act: -0.1700

Data Index: 668216 (C4 (Web Text))

Max Activating Token Index: 372

Text #5

Max Range: 3.8902. Min Range: -3.8902

Max Act: 3.7651. Min Act: -0.1700

Data Index: 748632 (C4 (Web Text))

Max Activating Token Index: 450

Text #6

Max Range: 3.8902. Min Range: -3.8902

Max Act: 3.7651. Min Act: -0.1700

Data Index: 535324 (C4 (Web Text))

Max Activating Token Index: 426

Text #7

Max Range: 3.8902. Min Range: -3.8902

Max Act: 3.7964. Min Act: -0.1700

Data Index: 286622 (C4 (Web Text))

Max Activating Token Index: 797

Text #8

Max Range: 3.8902. Min Range: -3.8902

Max Act: 3.7338. Min Act: -0.1700

Data Index: 590170 (C4 (Web Text))

Max Activating Token Index: 72

Text #9

Max Range: 3.8902. Min Range: -3.8902

Max Act: 3.7338. Min Act: -0.1700

Data Index: 368266 (C4 (Web Text))

Max Activating Token Index: 361

Text #10

Max Range: 3.8902. Min Range: -3.8902

Max Act: 3.7025. Min Act: -0.1700

Data Index: 803789 (C4 (Web Text))

Max Activating Token Index: 193

Text #11

Max Range: 3.8902. Min Range: -3.8902

Max Act: 3.7025. Min Act: -0.1700

Data Index: 896738 (C4 (Web Text))

Max Activating Token Index: 321

Text #12

Max Range: 3.8902. Min Range: -3.8902

Max Act: 3.7025. Min Act: -0.1700

Data Index: 154679 (C4 (Web Text))

Max Activating Token Index: 1016

Text #13

Max Range: 3.8902. Min Range: -3.8902

Max Act: 3.7025. Min Act: -0.1700

Data Index: 1134061 (C4 (Web Text))

Max Activating Token Index: 486

Text #14

Max Range: 3.8902. Min Range: -3.8902

Max Act: 3.7025. Min Act: -0.1700

Data Index: 1336898 (C4 (Web Text))

Max Activating Token Index: 142

Text #15

Max Range: 3.8902. Min Range: -3.8902

Max Act: 3.7025. Min Act: -0.1700

Data Index: 800104 (C4 (Web Text))

Max Activating Token Index: 788

Text #16

Max Range: 3.8902. Min Range: -3.8902

Max Act: 3.6712. Min Act: -0.1700

Data Index: 672035 (C4 (Web Text))

Max Activating Token Index: 950

Text #17

Max Range: 3.8902. Min Range: -3.8902

Max Act: 3.7025. Min Act: -0.1700

Data Index: 528576 (C4 (Web Text))

Max Activating Token Index: 223

Text #18

Max Range: 3.8902. Min Range: -3.8902

Max Act: 3.6712. Min Act: -0.1700

Data Index: 541540 (C4 (Web Text))

Max Activating Token Index: 390

Text #19

Max Range: 3.8902. Min Range: -3.8902

Max Act: 3.6712. Min Act: -0.1700

Data Index: 442163 (C4 (Web Text))

Max Activating Token Index: 245