Model: GELU Model: 1 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2018 in Layer 0

Load this data into an Interactive Neuroscope

See Documentation here

Hooked Transformer Loading: HookedTransformer.from_pretrained('gelu-1l')

Click the toggle to see the full text!

Text #0

Max Range: 3.9048. Min Range: -3.9048

Max Act: 3.9048. Min Act: -0.1627

Data Index: 1403546 (Python Code)

Max Activating Token Index: 511

Text #1

Max Range: 3.9048. Min Range: -3.9048

Max Act: 3.9003. Min Act: -0.1698

Data Index: 1642332 (Python Code)

Max Activating Token Index: 861

Text #2

Max Range: 3.9048. Min Range: -3.9048

Max Act: 3.8824. Min Act: -0.1699

Data Index: 144080 (C4 (Web Text))

Max Activating Token Index: 860

Text #3

Max Range: 3.9048. Min Range: -3.9048

Max Act: 3.8933. Min Act: -0.1695

Data Index: 1360986 (Python Code)

Max Activating Token Index: 958

Text #4

Max Range: 3.9048. Min Range: -3.9048

Max Act: 3.8757. Min Act: -0.1700

Data Index: 1083197 (C4 (Web Text))

Max Activating Token Index: 982

Text #5

Max Range: 3.9048. Min Range: -3.9048

Max Act: 3.8014. Min Act: -0.1700

Data Index: 1485889 (Python Code)

Max Activating Token Index: 59

Text #6

Max Range: 3.9048. Min Range: -3.9048

Max Act: 3.7861. Min Act: -0.1685

Data Index: 112429 (C4 (Web Text))

Max Activating Token Index: 168

Text #7

Max Range: 3.9048. Min Range: -3.9048

Max Act: 3.7387. Min Act: -0.1550

Data Index: 1230889 (C4 (Web Text))

Max Activating Token Index: 815

Text #8

Max Range: 3.9048. Min Range: -3.9048

Max Act: 3.7644. Min Act: -0.1698

Data Index: 1350407 (C4 (Web Text))

Max Activating Token Index: 741

Text #9

Max Range: 3.9048. Min Range: -3.9048

Max Act: 3.7523. Min Act: -0.1698

Data Index: 545584 (C4 (Web Text))

Max Activating Token Index: 645

Text #10

Max Range: 3.9048. Min Range: -3.9048

Max Act: 3.7166. Min Act: -0.1691

Data Index: 1549254 (Python Code)

Max Activating Token Index: 511

Text #11

Max Range: 3.9048. Min Range: -3.9048

Max Act: 3.7104. Min Act: -0.1697

Data Index: 1642386 (Python Code)

Max Activating Token Index: 875

Text #12

Max Range: 3.9048. Min Range: -3.9048

Max Act: 3.7199. Min Act: -0.1695

Data Index: 1161097 (C4 (Web Text))

Max Activating Token Index: 875

Text #13

Max Range: 3.9048. Min Range: -3.9048

Max Act: 3.6786. Min Act: -0.1580

Data Index: 599213 (C4 (Web Text))

Max Activating Token Index: 993

Text #14

Max Range: 3.9048. Min Range: -3.9048

Max Act: 3.7089. Min Act: -0.1700

Data Index: 1509038 (Python Code)

Max Activating Token Index: 818

Text #15

Max Range: 3.9048. Min Range: -3.9048

Max Act: 3.6991. Min Act: -0.1700

Data Index: 1618501 (Python Code)

Max Activating Token Index: 720

Text #16

Max Range: 3.9048. Min Range: -3.9048

Max Act: 3.6901. Min Act: -0.1579

Data Index: 1035998 (C4 (Web Text))

Max Activating Token Index: 152

Text #17

Max Range: 3.9048. Min Range: -3.9048

Max Act: 3.6834. Min Act: -0.1700

Data Index: 132678 (C4 (Web Text))

Max Activating Token Index: 109

Text #18

Max Range: 3.9048. Min Range: -3.9048

Max Act: 3.6529. Min Act: -0.1699

Data Index: 41232 (C4 (Web Text))

Max Activating Token Index: 112

Text #19

Max Range: 3.9048. Min Range: -3.9048

Max Act: 3.6715. Min Act: -0.1694

Data Index: 312792 (C4 (Web Text))

Max Activating Token Index: 47