Model: GELU Model: 1 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1729 in Layer 0

Load this data into an Interactive Neuroscope

See Documentation here

Hooked Transformer Loading: HookedTransformer.from_pretrained('gelu-1l')

Click the toggle to see the full text!

Text #0

Max Range: 2.3339. Min Range: -2.3339

Max Act: 2.3339. Min Act: -0.1700

Data Index: 1201235 (C4 (Web Text))

Max Activating Token Index: 901

Text #1

Max Range: 2.3339. Min Range: -2.3339

Max Act: 2.3016. Min Act: -0.1699

Data Index: 1452660 (Python Code)

Max Activating Token Index: 442

Text #2

Max Range: 2.3339. Min Range: -2.3339

Max Act: 2.3021. Min Act: -0.1699

Data Index: 712876 (C4 (Web Text))

Max Activating Token Index: 1023

Text #3

Max Range: 2.3339. Min Range: -2.3339

Max Act: 2.2643. Min Act: -0.1505

Data Index: 728907 (C4 (Web Text))

Max Activating Token Index: 829

Text #4

Max Range: 2.3339. Min Range: -2.3339

Max Act: 2.2594. Min Act: -0.1700

Data Index: 808833 (C4 (Web Text))

Max Activating Token Index: 919

Text #5

Max Range: 2.3339. Min Range: -2.3339

Max Act: 2.2498. Min Act: -0.1656

Data Index: 1483959 (Python Code)

Max Activating Token Index: 637

Text #6

Max Range: 2.3339. Min Range: -2.3339

Max Act: 2.2389. Min Act: -0.1700

Data Index: 745446 (C4 (Web Text))

Max Activating Token Index: 766

Text #7

Max Range: 2.3339. Min Range: -2.3339

Max Act: 2.2315. Min Act: -0.1700

Data Index: 1501484 (Python Code)

Max Activating Token Index: 567

Text #8

Max Range: 2.3339. Min Range: -2.3339

Max Act: 2.2400. Min Act: -0.1700

Data Index: 1388513 (Python Code)

Max Activating Token Index: 437

Text #9

Max Range: 2.3339. Min Range: -2.3339

Max Act: 2.2311. Min Act: -0.1700

Data Index: 1208546 (C4 (Web Text))

Max Activating Token Index: 1023

Text #10

Max Range: 2.3339. Min Range: -2.3339

Max Act: 2.2092. Min Act: -0.1700

Data Index: 1366804 (Python Code)

Max Activating Token Index: 323

Text #11

Max Range: 2.3339. Min Range: -2.3339

Max Act: 2.2114. Min Act: -0.1700

Data Index: 1580480 (Python Code)

Max Activating Token Index: 79

Text #12

Max Range: 2.3339. Min Range: -2.3339

Max Act: 2.1978. Min Act: -0.1699

Data Index: 1503043 (Python Code)

Max Activating Token Index: 976

Text #13

Max Range: 2.3339. Min Range: -2.3339

Max Act: 2.1964. Min Act: -0.1700

Data Index: 1372932 (Python Code)

Max Activating Token Index: 309

Text #14

Max Range: 2.3339. Min Range: -2.3339

Max Act: 2.2055. Min Act: -0.1697

Data Index: 631827 (C4 (Web Text))

Max Activating Token Index: 1023

Text #15

Max Range: 2.3339. Min Range: -2.3339

Max Act: 2.1802. Min Act: -0.1699

Data Index: 1654212 (Python Code)

Max Activating Token Index: 823

Text #16

Max Range: 2.3339. Min Range: -2.3339

Max Act: 2.1734. Min Act: -0.1699

Data Index: 247357 (C4 (Web Text))

Max Activating Token Index: 838

Text #17

Max Range: 2.3339. Min Range: -2.3339

Max Act: 2.1807. Min Act: -0.1700

Data Index: 1268236 (C4 (Web Text))

Max Activating Token Index: 437

Text #18

Max Range: 2.3339. Min Range: -2.3339

Max Act: 2.1707. Min Act: -0.1700

Data Index: 1460667 (Python Code)

Max Activating Token Index: 325

Text #19

Max Range: 2.3339. Min Range: -2.3339

Max Act: 2.1658. Min Act: -0.1699

Data Index: 1500322 (Python Code)

Max Activating Token Index: 348