Model: GELU Model: 1 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 776 in Layer 0

Load this data into an Interactive Neuroscope

See Documentation here

Hooked Transformer Loading: HookedTransformer.from_pretrained('gelu-1l')

Click the toggle to see the full text!

Text #0

Max Range: 3.9217. Min Range: -3.9217

Max Act: 3.9217. Min Act: -0.1665

Data Index: 1118609 (C4 (Web Text))

Max Activating Token Index: 930

Text #1

Max Range: 3.9217. Min Range: -3.9217

Max Act: 3.8626. Min Act: -0.1619

Data Index: 20070 (C4 (Web Text))

Max Activating Token Index: 278

Text #2

Max Range: 3.9217. Min Range: -3.9217

Max Act: 3.8655. Min Act: -0.1584

Data Index: 88657 (C4 (Web Text))

Max Activating Token Index: 599

Text #3

Max Range: 3.9217. Min Range: -3.9217

Max Act: 3.8397. Min Act: -0.1666

Data Index: 1575330 (Python Code)

Max Activating Token Index: 926

Text #4

Max Range: 3.9217. Min Range: -3.9217

Max Act: 3.7936. Min Act: -0.1684

Data Index: 979977 (C4 (Web Text))

Max Activating Token Index: 279

Text #5

Max Range: 3.9217. Min Range: -3.9217

Max Act: 3.7906. Min Act: -0.1699

Data Index: 1623090 (Python Code)

Max Activating Token Index: 405

Text #6

Max Range: 3.9217. Min Range: -3.9217

Max Act: 3.7963. Min Act: -0.1699

Data Index: 1293699 (C4 (Web Text))

Max Activating Token Index: 929

Text #7

Max Range: 3.9217. Min Range: -3.9217

Max Act: 3.7783. Min Act: -0.1699

Data Index: 190220 (C4 (Web Text))

Max Activating Token Index: 598

Text #8

Max Range: 3.9217. Min Range: -3.9217

Max Act: 3.7822. Min Act: -0.1643

Data Index: 1530843 (Python Code)

Max Activating Token Index: 486

Text #9

Max Range: 3.9217. Min Range: -3.9217

Max Act: 3.7767. Min Act: -0.1694

Data Index: 934145 (C4 (Web Text))

Max Activating Token Index: 600

Text #10

Max Range: 3.9217. Min Range: -3.9217

Max Act: 3.7733. Min Act: -0.1287

Data Index: 83466 (C4 (Web Text))

Max Activating Token Index: 929

Text #11

Max Range: 3.9217. Min Range: -3.9217

Max Act: 3.7715. Min Act: -0.1686

Data Index: 1165898 (C4 (Web Text))

Max Activating Token Index: 278

Text #12

Max Range: 3.9217. Min Range: -3.9217

Max Act: 3.7627. Min Act: -0.1678

Data Index: 464240 (C4 (Web Text))

Max Activating Token Index: 926

Text #13

Max Range: 3.9217. Min Range: -3.9217

Max Act: 3.7675. Min Act: -0.1699

Data Index: 386500 (C4 (Web Text))

Max Activating Token Index: 151

Text #14

Max Range: 3.9217. Min Range: -3.9217

Max Act: 3.7638. Min Act: -0.1697

Data Index: 861066 (C4 (Web Text))

Max Activating Token Index: 452

Text #15

Max Range: 3.9217. Min Range: -3.9217

Max Act: 3.7634. Min Act: -0.1690

Data Index: 1637556 (Python Code)

Max Activating Token Index: 574

Text #16

Max Range: 3.9217. Min Range: -3.9217

Max Act: 3.7647. Min Act: -0.1695

Data Index: 789157 (C4 (Web Text))

Max Activating Token Index: 193

Text #17

Max Range: 3.9217. Min Range: -3.9217

Max Act: 3.7424. Min Act: -0.1503

Data Index: 1186559 (C4 (Web Text))

Max Activating Token Index: 598

Text #18

Max Range: 3.9217. Min Range: -3.9217

Max Act: 3.7481. Min Act: -0.1644

Data Index: 1213106 (C4 (Web Text))

Max Activating Token Index: 927

Text #19

Max Range: 3.9217. Min Range: -3.9217

Max Act: 3.7408. Min Act: -0.1677

Data Index: 438196 (C4 (Web Text))

Max Activating Token Index: 581