Model: GELU Model: 1 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1028 in Layer 0

Load this data into an Interactive Neuroscope

See Documentation here

Hooked Transformer Loading: HookedTransformer.from_pretrained('gelu-1l')

Click the toggle to see the full text!

Text #0

Max Range: 3.6439. Min Range: -3.6439

Max Act: 3.6439. Min Act: -0.1700

Data Index: 1385037 (Python Code)

Max Activating Token Index: 764

Text #1

Max Range: 3.6439. Min Range: -3.6439

Max Act: 3.5881. Min Act: -0.1700

Data Index: 1492317 (Python Code)

Max Activating Token Index: 288

Text #2

Max Range: 3.6439. Min Range: -3.6439

Max Act: 3.5056. Min Act: -0.1700

Data Index: 493281 (C4 (Web Text))

Max Activating Token Index: 18

Text #3

Max Range: 3.6439. Min Range: -3.6439

Max Act: 3.5297. Min Act: -0.1700

Data Index: 1409479 (Python Code)

Max Activating Token Index: 84

Text #4

Max Range: 3.6439. Min Range: -3.6439

Max Act: 3.4555. Min Act: -0.1700

Data Index: 1234195 (C4 (Web Text))

Max Activating Token Index: 941

Text #5

Max Range: 3.6439. Min Range: -3.6439

Max Act: 3.4502. Min Act: -0.1700

Data Index: 1519776 (Python Code)

Max Activating Token Index: 575

Text #6

Max Range: 3.6439. Min Range: -3.6439

Max Act: 3.4245. Min Act: -0.1700

Data Index: 1469177 (Python Code)

Max Activating Token Index: 193

Text #7

Max Range: 3.6439. Min Range: -3.6439

Max Act: 3.4319. Min Act: -0.1700

Data Index: 681659 (C4 (Web Text))

Max Activating Token Index: 780

Text #8

Max Range: 3.6439. Min Range: -3.6439

Max Act: 3.4091. Min Act: -0.1700

Data Index: 1368783 (Python Code)

Max Activating Token Index: 219

Text #9

Max Range: 3.6439. Min Range: -3.6439

Max Act: 3.3995. Min Act: -0.1700

Data Index: 1122661 (C4 (Web Text))

Max Activating Token Index: 523

Text #10

Max Range: 3.6439. Min Range: -3.6439

Max Act: 3.4020. Min Act: -0.1700

Data Index: 1227779 (C4 (Web Text))

Max Activating Token Index: 403

Text #11

Max Range: 3.6439. Min Range: -3.6439

Max Act: 3.3487. Min Act: -0.1700

Data Index: 129759 (C4 (Web Text))

Max Activating Token Index: 241

Text #12

Max Range: 3.6439. Min Range: -3.6439

Max Act: 3.3750. Min Act: -0.1700

Data Index: 1381447 (Python Code)

Max Activating Token Index: 605

Text #13

Max Range: 3.6439. Min Range: -3.6439

Max Act: 3.3653. Min Act: -0.1684

Data Index: 830474 (C4 (Web Text))

Max Activating Token Index: 681

Text #14

Max Range: 3.6439. Min Range: -3.6439

Max Act: 3.3199. Min Act: -0.1700

Data Index: 1068648 (C4 (Web Text))

Max Activating Token Index: 138

Text #15

Max Range: 3.6439. Min Range: -3.6439

Max Act: 3.2956. Min Act: -0.1700

Data Index: 1429412 (Python Code)

Max Activating Token Index: 974

Text #16

Max Range: 3.6439. Min Range: -3.6439

Max Act: 3.2861. Min Act: -0.1700

Data Index: 1583658 (Python Code)

Max Activating Token Index: 774

Text #17

Max Range: 3.6439. Min Range: -3.6439

Max Act: 3.3074. Min Act: -0.1700

Data Index: 1596389 (Python Code)

Max Activating Token Index: 804

Text #18

Max Range: 3.6439. Min Range: -3.6439

Max Act: 3.3021. Min Act: -0.1700

Data Index: 1427704 (Python Code)

Max Activating Token Index: 866

Text #19

Max Range: 3.6439. Min Range: -3.6439

Max Act: 3.2915. Min Act: -0.1700

Data Index: 162606 (C4 (Web Text))

Max Activating Token Index: 613