Model: GELU Model: 2 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 990 in Layer 1

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('gelu-2l')

Text #0

Max Range: 4.1914. Min Range: -4.1914

Max Act: 4.1914. Min Act: -0.1700

Data Index: 1526761 (Python Code)

Max Activating Token Index: 559

Text #1

Max Range: 4.1914. Min Range: -4.1914

Max Act: 4.1601. Min Act: -0.1700

Data Index: 1533243 (Python Code)

Max Activating Token Index: 909

Text #2

Max Range: 4.1914. Min Range: -4.1914

Max Act: 4.0663. Min Act: -0.1700

Data Index: 1402490 (Python Code)

Max Activating Token Index: 368

Text #3

Max Range: 4.1914. Min Range: -4.1914

Max Act: 3.9882. Min Act: -0.1700

Data Index: 1616541 (Python Code)

Max Activating Token Index: 577

Text #4

Max Range: 4.1914. Min Range: -4.1914

Max Act: 3.9569. Min Act: -0.1700

Data Index: 1495626 (Python Code)

Max Activating Token Index: 199

Text #5

Max Range: 4.1914. Min Range: -4.1914

Max Act: 3.9960. Min Act: -0.1700

Data Index: 1582974 (Python Code)

Max Activating Token Index: 552

Text #6

Max Range: 4.1914. Min Range: -4.1914

Max Act: 3.9647. Min Act: -0.1700

Data Index: 1510542 (Python Code)

Max Activating Token Index: 767

Text #7

Max Range: 4.1914. Min Range: -4.1914

Max Act: 3.9647. Min Act: -0.1700

Data Index: 1420116 (Python Code)

Max Activating Token Index: 535

Text #8

Max Range: 4.1914. Min Range: -4.1914

Max Act: 3.9569. Min Act: -0.1700

Data Index: 1614519 (Python Code)

Max Activating Token Index: 616

Text #9

Max Range: 4.1914. Min Range: -4.1914

Max Act: 3.9413. Min Act: -0.1700

Data Index: 1422311 (Python Code)

Max Activating Token Index: 1009

Text #10

Max Range: 4.1914. Min Range: -4.1914

Max Act: 3.9100. Min Act: -0.1700

Data Index: 1543962 (Python Code)

Max Activating Token Index: 322

Text #11

Max Range: 4.1914. Min Range: -4.1914

Max Act: 3.8944. Min Act: -0.1700

Data Index: 1445352 (Python Code)

Max Activating Token Index: 719

Text #12

Max Range: 4.1914. Min Range: -4.1914

Max Act: 3.9413. Min Act: -0.1700

Data Index: 1503370 (Python Code)

Max Activating Token Index: 900

Text #13

Max Range: 4.1914. Min Range: -4.1914

Max Act: 3.8865. Min Act: -0.1700

Data Index: 1446925 (Python Code)

Max Activating Token Index: 306

Text #14

Max Range: 4.1914. Min Range: -4.1914

Max Act: 3.9725. Min Act: -0.1700

Data Index: 1581317 (Python Code)

Max Activating Token Index: 988

Text #15

Max Range: 4.1914. Min Range: -4.1914

Max Act: 3.9178. Min Act: -0.1700

Data Index: 1620865 (Python Code)

Max Activating Token Index: 804

Text #16

Max Range: 4.1914. Min Range: -4.1914

Max Act: 3.8944. Min Act: -0.1700

Data Index: 1589448 (Python Code)

Max Activating Token Index: 23

Text #17

Max Range: 4.1914. Min Range: -4.1914

Max Act: 3.8865. Min Act: -0.1700

Data Index: 1633678 (Python Code)

Max Activating Token Index: 791

Text #18

Max Range: 4.1914. Min Range: -4.1914

Max Act: 3.8787. Min Act: -0.1700

Data Index: 1512559 (Python Code)

Max Activating Token Index: 486

Text #19

Max Range: 4.1914. Min Range: -4.1914

Max Act: 3.8865. Min Act: -0.1700

Data Index: 1583000 (Python Code)

Max Activating Token Index: 791