Model: GELU Model: 1 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 603 in Layer 0

Load this data into an Interactive Neuroscope

See Documentation here

Hooked Transformer Loading: HookedTransformer.from_pretrained('gelu-1l')

Click the toggle to see the full text!

Text #0

Max Range: 4.7121. Min Range: -4.7121

Max Act: 4.7121. Min Act: -0.1700

Data Index: 288267 (C4 (Web Text))

Max Activating Token Index: 861

Text #1

Max Range: 4.7121. Min Range: -4.7121

Max Act: 4.6418. Min Act: -0.1700

Data Index: 755596 (C4 (Web Text))

Max Activating Token Index: 447

Text #2

Max Range: 4.7121. Min Range: -4.7121

Max Act: 4.5929. Min Act: -0.1700

Data Index: 1440219 (Python Code)

Max Activating Token Index: 490

Text #3

Max Range: 4.7121. Min Range: -4.7121

Max Act: 4.5939. Min Act: -0.1700

Data Index: 307122 (C4 (Web Text))

Max Activating Token Index: 833

Text #4

Max Range: 4.7121. Min Range: -4.7121

Max Act: 4.6155. Min Act: -0.1699

Data Index: 64200 (C4 (Web Text))

Max Activating Token Index: 818

Text #5

Max Range: 4.7121. Min Range: -4.7121

Max Act: 4.5533. Min Act: -0.1700

Data Index: 355404 (C4 (Web Text))

Max Activating Token Index: 722

Text #6

Max Range: 4.7121. Min Range: -4.7121

Max Act: 4.5263. Min Act: -0.1700

Data Index: 1637215 (Python Code)

Max Activating Token Index: 899

Text #7

Max Range: 4.7121. Min Range: -4.7121

Max Act: 4.4825. Min Act: -0.1700

Data Index: 1507605 (Python Code)

Max Activating Token Index: 590

Text #8

Max Range: 4.7121. Min Range: -4.7121

Max Act: 4.4860. Min Act: -0.1700

Data Index: 1518659 (Python Code)

Max Activating Token Index: 572

Text #9

Max Range: 4.7121. Min Range: -4.7121

Max Act: 4.5421. Min Act: -0.1700

Data Index: 1619585 (Python Code)

Max Activating Token Index: 977

Text #10

Max Range: 4.7121. Min Range: -4.7121

Max Act: 4.4498. Min Act: -0.1699

Data Index: 1582218 (Python Code)

Max Activating Token Index: 509

Text #11

Max Range: 4.7121. Min Range: -4.7121

Max Act: 4.4797. Min Act: -0.1700

Data Index: 374778 (C4 (Web Text))

Max Activating Token Index: 948

Text #12

Max Range: 4.7121. Min Range: -4.7121

Max Act: 4.4309. Min Act: -0.1700

Data Index: 356712 (C4 (Web Text))

Max Activating Token Index: 862

Text #13

Max Range: 4.7121. Min Range: -4.7121

Max Act: 4.4193. Min Act: -0.1700

Data Index: 611991 (C4 (Web Text))

Max Activating Token Index: 483

Text #14

Max Range: 4.7121. Min Range: -4.7121

Max Act: 4.4292. Min Act: -0.1683

Data Index: 1288361 (C4 (Web Text))

Max Activating Token Index: 529

Text #15

Max Range: 4.7121. Min Range: -4.7121

Max Act: 4.4066. Min Act: -0.1700

Data Index: 1645531 (Python Code)

Max Activating Token Index: 237

Text #16

Max Range: 4.7121. Min Range: -4.7121

Max Act: 4.3759. Min Act: -0.1699

Data Index: 1389827 (Python Code)

Max Activating Token Index: 662

Text #17

Max Range: 4.7121. Min Range: -4.7121

Max Act: 4.4132. Min Act: -0.1700

Data Index: 1454540 (Python Code)

Max Activating Token Index: 405

Text #18

Max Range: 4.7121. Min Range: -4.7121

Max Act: 4.3958. Min Act: -0.1700

Data Index: 602870 (C4 (Web Text))

Max Activating Token Index: 404

Text #19

Max Range: 4.7121. Min Range: -4.7121

Max Act: 4.4010. Min Act: -0.1700

Data Index: 1185759 (C4 (Web Text))

Max Activating Token Index: 654