Model: GELU Model: 1 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 501 in Layer 0

Load this data into an Interactive Neuroscope

See Documentation here

Hooked Transformer Loading: HookedTransformer.from_pretrained('gelu-1l')

Click the toggle to see the full text!

Text #0

Max Range: 2.8125. Min Range: -2.8125

Max Act: 2.8125. Min Act: -0.1700

Data Index: 894997 (C4 (Web Text))

Max Activating Token Index: 327

Text #1

Max Range: 2.8125. Min Range: -2.8125

Max Act: 2.7954. Min Act: -0.1698

Data Index: 1568701 (Python Code)

Max Activating Token Index: 1009

Text #2

Max Range: 2.8125. Min Range: -2.8125

Max Act: 2.7537. Min Act: -0.1700

Data Index: 573610 (C4 (Web Text))

Max Activating Token Index: 765

Text #3

Max Range: 2.8125. Min Range: -2.8125

Max Act: 2.7305. Min Act: -0.1700

Data Index: 58360 (C4 (Web Text))

Max Activating Token Index: 354

Text #4

Max Range: 2.8125. Min Range: -2.8125

Max Act: 2.7183. Min Act: -0.1700

Data Index: 149842 (C4 (Web Text))

Max Activating Token Index: 909

Text #5

Max Range: 2.8125. Min Range: -2.8125

Max Act: 2.7216. Min Act: -0.1700

Data Index: 1597668 (Python Code)

Max Activating Token Index: 695

Text #6

Max Range: 2.8125. Min Range: -2.8125

Max Act: 2.7087. Min Act: -0.1700

Data Index: 1227867 (C4 (Web Text))

Max Activating Token Index: 151

Text #7

Max Range: 2.8125. Min Range: -2.8125

Max Act: 2.6989. Min Act: -0.1700

Data Index: 117501 (C4 (Web Text))

Max Activating Token Index: 436

Text #8

Max Range: 2.8125. Min Range: -2.8125

Max Act: 2.6813. Min Act: -0.1700

Data Index: 757482 (C4 (Web Text))

Max Activating Token Index: 160

Text #9

Max Range: 2.8125. Min Range: -2.8125

Max Act: 2.6749. Min Act: -0.1700

Data Index: 793409 (C4 (Web Text))

Max Activating Token Index: 464

Text #10

Max Range: 2.8125. Min Range: -2.8125

Max Act: 2.6693. Min Act: -0.1700

Data Index: 1522543 (Python Code)

Max Activating Token Index: 788

Text #11

Max Range: 2.8125. Min Range: -2.8125

Max Act: 2.6853. Min Act: -0.1700

Data Index: 381979 (C4 (Web Text))

Max Activating Token Index: 637

Text #12

Max Range: 2.8125. Min Range: -2.8125

Max Act: 2.6724. Min Act: -0.1699

Data Index: 1642154 (Python Code)

Max Activating Token Index: 93

Text #13

Max Range: 2.8125. Min Range: -2.8125

Max Act: 2.6658. Min Act: -0.1699

Data Index: 1421796 (Python Code)

Max Activating Token Index: 994

Text #14

Max Range: 2.8125. Min Range: -2.8125

Max Act: 2.6508. Min Act: -0.1700

Data Index: 11925 (C4 (Web Text))

Max Activating Token Index: 199

Text #15

Max Range: 2.8125. Min Range: -2.8125

Max Act: 2.6589. Min Act: -0.1700

Data Index: 226707 (C4 (Web Text))

Max Activating Token Index: 20

Text #16

Max Range: 2.8125. Min Range: -2.8125

Max Act: 2.6406. Min Act: -0.1700

Data Index: 1262449 (C4 (Web Text))

Max Activating Token Index: 403

Text #17

Max Range: 2.8125. Min Range: -2.8125

Max Act: 2.6477. Min Act: -0.1700

Data Index: 536492 (C4 (Web Text))

Max Activating Token Index: 876

Text #18

Max Range: 2.8125. Min Range: -2.8125

Max Act: 2.6552. Min Act: -0.1700

Data Index: 375042 (C4 (Web Text))

Max Activating Token Index: 589

Text #19

Max Range: 2.8125. Min Range: -2.8125

Max Act: 2.6458. Min Act: -0.1700

Data Index: 1540911 (Python Code)

Max Activating Token Index: 272