Model: GELU Model: 1 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 696 in Layer 0

Load this data into an Interactive Neuroscope

See Documentation here

Hooked Transformer Loading: HookedTransformer.from_pretrained('gelu-1l')

Click the toggle to see the full text!

Text #0

Max Range: 2.9999. Min Range: -2.9999

Max Act: 2.9999. Min Act: -0.1700

Data Index: 1343754 (C4 (Web Text))

Max Activating Token Index: 522

Text #1

Max Range: 2.9999. Min Range: -2.9999

Max Act: 2.9666. Min Act: -0.1700

Data Index: 331633 (C4 (Web Text))

Max Activating Token Index: 689

Text #2

Max Range: 2.9999. Min Range: -2.9999

Max Act: 2.9515. Min Act: -0.1700

Data Index: 392938 (C4 (Web Text))

Max Activating Token Index: 839

Text #3

Max Range: 2.9999. Min Range: -2.9999

Max Act: 2.8739. Min Act: -0.1700

Data Index: 1412238 (Python Code)

Max Activating Token Index: 5

Text #4

Max Range: 2.9999. Min Range: -2.9999

Max Act: 2.8900. Min Act: -0.1700

Data Index: 308688 (C4 (Web Text))

Max Activating Token Index: 205

Text #5

Max Range: 2.9999. Min Range: -2.9999

Max Act: 2.8670. Min Act: -0.1700

Data Index: 1104518 (C4 (Web Text))

Max Activating Token Index: 594

Text #6

Max Range: 2.9999. Min Range: -2.9999

Max Act: 2.8667. Min Act: -0.1700

Data Index: 62151 (C4 (Web Text))

Max Activating Token Index: 84

Text #7

Max Range: 2.9999. Min Range: -2.9999

Max Act: 2.8647. Min Act: -0.1700

Data Index: 123481 (C4 (Web Text))

Max Activating Token Index: 774

Text #8

Max Range: 2.9999. Min Range: -2.9999

Max Act: 2.8590. Min Act: -0.1700

Data Index: 250754 (C4 (Web Text))

Max Activating Token Index: 955

Text #9

Max Range: 2.9999. Min Range: -2.9999

Max Act: 2.8725. Min Act: -0.1700

Data Index: 1003419 (C4 (Web Text))

Max Activating Token Index: 145

Text #10

Max Range: 2.9999. Min Range: -2.9999

Max Act: 2.8619. Min Act: -0.1700

Data Index: 730988 (C4 (Web Text))

Max Activating Token Index: 685

Text #11

Max Range: 2.9999. Min Range: -2.9999

Max Act: 2.8695. Min Act: -0.1700

Data Index: 1483256 (Python Code)

Max Activating Token Index: 721

Text #12

Max Range: 2.9999. Min Range: -2.9999

Max Act: 2.8582. Min Act: -0.1700

Data Index: 343248 (C4 (Web Text))

Max Activating Token Index: 685

Text #13

Max Range: 2.9999. Min Range: -2.9999

Max Act: 2.8652. Min Act: -0.1700

Data Index: 871718 (C4 (Web Text))

Max Activating Token Index: 218

Text #14

Max Range: 2.9999. Min Range: -2.9999

Max Act: 2.8568. Min Act: -0.1700

Data Index: 1080396 (C4 (Web Text))

Max Activating Token Index: 296

Text #15

Max Range: 2.9999. Min Range: -2.9999

Max Act: 2.8587. Min Act: -0.1700

Data Index: 1007322 (C4 (Web Text))

Max Activating Token Index: 8

Text #16

Max Range: 2.9999. Min Range: -2.9999

Max Act: 2.8365. Min Act: -0.1700

Data Index: 808810 (C4 (Web Text))

Max Activating Token Index: 833

Text #17

Max Range: 2.9999. Min Range: -2.9999

Max Act: 2.8657. Min Act: -0.1700

Data Index: 1471923 (Python Code)

Max Activating Token Index: 91

Text #18

Max Range: 2.9999. Min Range: -2.9999

Max Act: 2.8465. Min Act: -0.1700

Data Index: 244091 (C4 (Web Text))

Max Activating Token Index: 931

Text #19

Max Range: 2.9999. Min Range: -2.9999

Max Act: 2.8427. Min Act: -0.1700

Data Index: 1560723 (Python Code)

Max Activating Token Index: 326