Model: GELU Model: 4 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1429 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('gelu-4l')

Text #0

Max Range: 3.2967. Min Range: -3.2967

Max Act: 3.2967. Min Act: -0.1700

Data Index: 1637782 (Python Code)

Max Activating Token Index: 670

Text #1

Max Range: 3.2967. Min Range: -3.2967

Max Act: 3.2496. Min Act: -0.1700

Data Index: 1487634 (Python Code)

Max Activating Token Index: 262

Text #2

Max Range: 3.2967. Min Range: -3.2967

Max Act: 3.2339. Min Act: -0.1700

Data Index: 1372580 (Python Code)

Max Activating Token Index: 882

Text #3

Max Range: 3.2967. Min Range: -3.2967

Max Act: 3.2496. Min Act: -0.1700

Data Index: 1444823 (Python Code)

Max Activating Token Index: 315

Text #4

Max Range: 3.2967. Min Range: -3.2967

Max Act: 3.1395. Min Act: -0.1700

Data Index: 1497063 (Python Code)

Max Activating Token Index: 602

Text #5

Max Range: 3.2967. Min Range: -3.2967

Max Act: 3.1867. Min Act: -0.1700

Data Index: 1431890 (Python Code)

Max Activating Token Index: 906

Text #6

Max Range: 3.2967. Min Range: -3.2967

Max Act: 3.1395. Min Act: -0.1700

Data Index: 1655480 (Python Code)

Max Activating Token Index: 598

Text #7

Max Range: 3.2967. Min Range: -3.2967

Max Act: 3.1552. Min Act: -0.1700

Data Index: 1397428 (Python Code)

Max Activating Token Index: 241

Text #8

Max Range: 3.2967. Min Range: -3.2967

Max Act: 3.1395. Min Act: -0.1700

Data Index: 1577794 (Python Code)

Max Activating Token Index: 376

Text #9

Max Range: 3.2967. Min Range: -3.2967

Max Act: 3.1395. Min Act: -0.1700

Data Index: 1539009 (Python Code)

Max Activating Token Index: 60

Text #10

Max Range: 3.2967. Min Range: -3.2967

Max Act: 3.1237. Min Act: -0.1700

Data Index: 1471248 (Python Code)

Max Activating Token Index: 235

Text #11

Max Range: 3.2967. Min Range: -3.2967

Max Act: 3.1395. Min Act: -0.1700

Data Index: 1363061 (Python Code)

Max Activating Token Index: 448

Text #12

Max Range: 3.2967. Min Range: -3.2967

Max Act: 3.1552. Min Act: -0.1700

Data Index: 1409543 (Python Code)

Max Activating Token Index: 452

Text #13

Max Range: 3.2967. Min Range: -3.2967

Max Act: 3.1237. Min Act: -0.1700

Data Index: 1390029 (Python Code)

Max Activating Token Index: 176

Text #14

Max Range: 3.2967. Min Range: -3.2967

Max Act: 3.1237. Min Act: -0.1700

Data Index: 1463376 (Python Code)

Max Activating Token Index: 115

Text #15

Max Range: 3.2967. Min Range: -3.2967

Max Act: 3.1237. Min Act: -0.1700

Data Index: 1582541 (Python Code)

Max Activating Token Index: 296

Text #16

Max Range: 3.2967. Min Range: -3.2967

Max Act: 3.1395. Min Act: -0.1700

Data Index: 1586063 (Python Code)

Max Activating Token Index: 827

Text #17

Max Range: 3.2967. Min Range: -3.2967

Max Act: 3.0922. Min Act: -0.1700

Data Index: 1422144 (Python Code)

Max Activating Token Index: 65

Text #18

Max Range: 3.2967. Min Range: -3.2967

Max Act: 3.1395. Min Act: -0.1700

Data Index: 1632138 (Python Code)

Max Activating Token Index: 396

Text #19

Max Range: 3.2967. Min Range: -3.2967

Max Act: 3.1080. Min Act: -0.1700

Data Index: 1481864 (Python Code)

Max Activating Token Index: 88