Model: GELU Model: 1 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 638 in Layer 0

Load this data into an Interactive Neuroscope

See Documentation here

Hooked Transformer Loading: HookedTransformer.from_pretrained('gelu-1l')

Click the toggle to see the full text!

Text #0

Max Range: 3.7688. Min Range: -3.7688

Max Act: 3.7688. Min Act: -0.1700

Data Index: 1578779 (Python Code)

Max Activating Token Index: 600

Text #1

Max Range: 3.7688. Min Range: -3.7688

Max Act: 3.7684. Min Act: -0.1696

Data Index: 1412168 (Python Code)

Max Activating Token Index: 25

Text #2

Max Range: 3.7688. Min Range: -3.7688

Max Act: 3.7305. Min Act: -0.1699

Data Index: 744664 (C4 (Web Text))

Max Activating Token Index: 992

Text #3

Max Range: 3.7688. Min Range: -3.7688

Max Act: 3.6823. Min Act: -0.1697

Data Index: 1312982 (C4 (Web Text))

Max Activating Token Index: 417

Text #4

Max Range: 3.7688. Min Range: -3.7688

Max Act: 3.6303. Min Act: -0.1697

Data Index: 1241820 (C4 (Web Text))

Max Activating Token Index: 464

Text #5

Max Range: 3.7688. Min Range: -3.7688

Max Act: 3.6011. Min Act: -0.1698

Data Index: 1462564 (Python Code)

Max Activating Token Index: 212

Text #6

Max Range: 3.7688. Min Range: -3.7688

Max Act: 3.5842. Min Act: -0.1700

Data Index: 1192016 (C4 (Web Text))

Max Activating Token Index: 243

Text #7

Max Range: 3.7688. Min Range: -3.7688

Max Act: 3.5471. Min Act: -0.1536

Data Index: 77548 (C4 (Web Text))

Max Activating Token Index: 875

Text #8

Max Range: 3.7688. Min Range: -3.7688

Max Act: 3.5602. Min Act: -0.1431

Data Index: 1629911 (Python Code)

Max Activating Token Index: 452

Text #9

Max Range: 3.7688. Min Range: -3.7688

Max Act: 3.5528. Min Act: -0.1697

Data Index: 1545328 (Python Code)

Max Activating Token Index: 514

Text #10

Max Range: 3.7688. Min Range: -3.7688

Max Act: 3.5223. Min Act: -0.1502

Data Index: 26113 (C4 (Web Text))

Max Activating Token Index: 425

Text #11

Max Range: 3.7688. Min Range: -3.7688

Max Act: 3.5071. Min Act: -0.1615

Data Index: 1493691 (Python Code)

Max Activating Token Index: 1004

Text #12

Max Range: 3.7688. Min Range: -3.7688

Max Act: 3.4876. Min Act: -0.1700

Data Index: 1569459 (Python Code)

Max Activating Token Index: 607

Text #13

Max Range: 3.7688. Min Range: -3.7688

Max Act: 3.4974. Min Act: -0.1698

Data Index: 1390307 (Python Code)

Max Activating Token Index: 459

Text #14

Max Range: 3.7688. Min Range: -3.7688

Max Act: 3.5024. Min Act: -0.1699

Data Index: 990730 (C4 (Web Text))

Max Activating Token Index: 906

Text #15

Max Range: 3.7688. Min Range: -3.7688

Max Act: 3.4989. Min Act: -0.1695

Data Index: 1063871 (C4 (Web Text))

Max Activating Token Index: 989

Text #16

Max Range: 3.7688. Min Range: -3.7688

Max Act: 3.4890. Min Act: -0.1688

Data Index: 248885 (C4 (Web Text))

Max Activating Token Index: 765

Text #17

Max Range: 3.7688. Min Range: -3.7688

Max Act: 3.4932. Min Act: -0.1700

Data Index: 1273011 (C4 (Web Text))

Max Activating Token Index: 694

Text #18

Max Range: 3.7688. Min Range: -3.7688

Max Act: 3.4580. Min Act: -0.1699

Data Index: 1161570 (C4 (Web Text))

Max Activating Token Index: 251

Text #19

Max Range: 3.7688. Min Range: -3.7688

Max Act: 3.4521. Min Act: -0.1690

Data Index: 1511924 (Python Code)

Max Activating Token Index: 597