Model: GELU Model: 2 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1799 in Layer 1

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('gelu-2l')

Text #0

Max Range: 5.1301. Min Range: -5.1301

Max Act: 5.1301. Min Act: -0.1700

Data Index: 1462459 (Python Code)

Max Activating Token Index: 690

Text #1

Max Range: 5.1301. Min Range: -5.1301

Max Act: 5.0676. Min Act: -0.1700

Data Index: 1456061 (Python Code)

Max Activating Token Index: 1015

Text #2

Max Range: 5.1301. Min Range: -5.1301

Max Act: 5.0051. Min Act: -0.1700

Data Index: 1540054 (Python Code)

Max Activating Token Index: 949

Text #3

Max Range: 5.1301. Min Range: -5.1301

Max Act: 5.0364. Min Act: -0.1700

Data Index: 1588207 (Python Code)

Max Activating Token Index: 782

Text #4

Max Range: 5.1301. Min Range: -5.1301

Max Act: 4.9739. Min Act: -0.1700

Data Index: 1653070 (Python Code)

Max Activating Token Index: 342

Text #5

Max Range: 5.1301. Min Range: -5.1301

Max Act: 4.9739. Min Act: -0.1700

Data Index: 1544933 (Python Code)

Max Activating Token Index: 772

Text #6

Max Range: 5.1301. Min Range: -5.1301

Max Act: 4.9739. Min Act: -0.1700

Data Index: 1430541 (Python Code)

Max Activating Token Index: 1003

Text #7

Max Range: 5.1301. Min Range: -5.1301

Max Act: 4.9426. Min Act: -0.1700

Data Index: 1401872 (Python Code)

Max Activating Token Index: 977

Text #8

Max Range: 5.1301. Min Range: -5.1301

Max Act: 4.9739. Min Act: -0.1700

Data Index: 1589934 (Python Code)

Max Activating Token Index: 299

Text #9

Max Range: 5.1301. Min Range: -5.1301

Max Act: 4.9426. Min Act: -0.1700

Data Index: 1475586 (Python Code)

Max Activating Token Index: 726

Text #10

Max Range: 5.1301. Min Range: -5.1301

Max Act: 4.9114. Min Act: -0.1700

Data Index: 1526152 (Python Code)

Max Activating Token Index: 601

Text #11

Max Range: 5.1301. Min Range: -5.1301

Max Act: 4.9426. Min Act: -0.1700

Data Index: 1411441 (Python Code)

Max Activating Token Index: 688

Text #12

Max Range: 5.1301. Min Range: -5.1301

Max Act: 4.9114. Min Act: -0.1700

Data Index: 1566332 (Python Code)

Max Activating Token Index: 789

Text #13

Max Range: 5.1301. Min Range: -5.1301

Max Act: 4.8801. Min Act: -0.1700

Data Index: 1527113 (Python Code)

Max Activating Token Index: 923

Text #14

Max Range: 5.1301. Min Range: -5.1301

Max Act: 4.8801. Min Act: -0.1700

Data Index: 1565785 (Python Code)

Max Activating Token Index: 833

Text #15

Max Range: 5.1301. Min Range: -5.1301

Max Act: 4.9114. Min Act: -0.1700

Data Index: 1606509 (Python Code)

Max Activating Token Index: 835

Text #16

Max Range: 5.1301. Min Range: -5.1301

Max Act: 4.8801. Min Act: -0.1700

Data Index: 1536751 (Python Code)

Max Activating Token Index: 744

Text #17

Max Range: 5.1301. Min Range: -5.1301

Max Act: 4.9426. Min Act: -0.1700

Data Index: 1592622 (Python Code)

Max Activating Token Index: 347

Text #18

Max Range: 5.1301. Min Range: -5.1301

Max Act: 4.8801. Min Act: -0.1700

Data Index: 1370919 (Python Code)

Max Activating Token Index: 686

Text #19

Max Range: 5.1301. Min Range: -5.1301

Max Act: 4.8801. Min Act: -0.1700

Data Index: 1519702 (Python Code)

Max Activating Token Index: 850