Model: GELU Model: 4 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 766 in Layer 0

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('gelu-4l')

Text #0

Max Range: 3.0472. Min Range: -3.0472

Max Act: 3.0472. Min Act: -0.1697

Data Index: 159547 (C4 (Web Text))

Max Activating Token Index: 708

Text #1

Max Range: 3.0472. Min Range: -3.0472

Max Act: 3.0156. Min Act: -0.1693

Data Index: 462617 (C4 (Web Text))

Max Activating Token Index: 754

Text #2

Max Range: 3.0472. Min Range: -3.0472

Max Act: 2.9998. Min Act: -0.1694

Data Index: 863936 (C4 (Web Text))

Max Activating Token Index: 858

Text #3

Max Range: 3.0472. Min Range: -3.0472

Max Act: 2.9998. Min Act: -0.1690

Data Index: 1146670 (C4 (Web Text))

Max Activating Token Index: 803

Text #4

Max Range: 3.0472. Min Range: -3.0472

Max Act: 2.9998. Min Act: -0.1669

Data Index: 1328154 (C4 (Web Text))

Max Activating Token Index: 708

Text #5

Max Range: 3.0472. Min Range: -3.0472

Max Act: 2.9840. Min Act: -0.1685

Data Index: 292773 (C4 (Web Text))

Max Activating Token Index: 747

Text #6

Max Range: 3.0472. Min Range: -3.0472

Max Act: 2.9998. Min Act: -0.1639

Data Index: 619510 (C4 (Web Text))

Max Activating Token Index: 754

Text #7

Max Range: 3.0472. Min Range: -3.0472

Max Act: 2.9840. Min Act: -0.1700

Data Index: 667397 (C4 (Web Text))

Max Activating Token Index: 752

Text #8

Max Range: 3.0472. Min Range: -3.0472

Max Act: 2.9840. Min Act: -0.1659

Data Index: 1095653 (C4 (Web Text))

Max Activating Token Index: 708

Text #9

Max Range: 3.0472. Min Range: -3.0472

Max Act: 2.9998. Min Act: -0.1692

Data Index: 727548 (C4 (Web Text))

Max Activating Token Index: 887

Text #10

Max Range: 3.0472. Min Range: -3.0472

Max Act: 2.9840. Min Act: -0.1699

Data Index: 813138 (C4 (Web Text))

Max Activating Token Index: 917

Text #11

Max Range: 3.0472. Min Range: -3.0472

Max Act: 2.9840. Min Act: -0.1658

Data Index: 111237 (C4 (Web Text))

Max Activating Token Index: 882

Text #12

Max Range: 3.0472. Min Range: -3.0472

Max Act: 2.9840. Min Act: -0.1652

Data Index: 467262 (C4 (Web Text))

Max Activating Token Index: 858

Text #13

Max Range: 3.0472. Min Range: -3.0472

Max Act: 2.9682. Min Act: -0.1653

Data Index: 1208100 (C4 (Web Text))

Max Activating Token Index: 741

Text #14

Max Range: 3.0472. Min Range: -3.0472

Max Act: 2.9682. Min Act: -0.1658

Data Index: 1258102 (C4 (Web Text))

Max Activating Token Index: 752

Text #15

Max Range: 3.0472. Min Range: -3.0472

Max Act: 2.9682. Min Act: -0.1662

Data Index: 160532 (C4 (Web Text))

Max Activating Token Index: 544

Text #16

Max Range: 3.0472. Min Range: -3.0472

Max Act: 2.9524. Min Act: -0.1639

Data Index: 533610 (C4 (Web Text))

Max Activating Token Index: 383

Text #17

Max Range: 3.0472. Min Range: -3.0472

Max Act: 2.9524. Min Act: -0.1699

Data Index: 1029896 (C4 (Web Text))

Max Activating Token Index: 652

Text #18

Max Range: 3.0472. Min Range: -3.0472

Max Act: 2.9524. Min Act: -0.1699

Data Index: 265194 (C4 (Web Text))

Max Activating Token Index: 845

Text #19

Max Range: 3.0472. Min Range: -3.0472

Max Act: 2.9524. Min Act: -0.1639

Data Index: 594467 (C4 (Web Text))

Max Activating Token Index: 498