Model: GELU Model: 4 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1200 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('gelu-4l')

Text #0

Max Range: 5.9117. Min Range: -5.9117

Max Act: 5.8492. Min Act: -0.1700

Data Index: 1529016 (Python Code)

Max Activating Token Index: 137

Text #1

Max Range: 5.9117. Min Range: -5.9117

Max Act: 5.9117. Min Act: -0.1700

Data Index: 1577880 (Python Code)

Max Activating Token Index: 375

Text #2

Max Range: 5.9117. Min Range: -5.9117

Max Act: 5.7242. Min Act: -0.1700

Data Index: 1584561 (Python Code)

Max Activating Token Index: 68

Text #3

Max Range: 5.9117. Min Range: -5.9117

Max Act: 5.7555. Min Act: -0.1700

Data Index: 1651643 (Python Code)

Max Activating Token Index: 687

Text #4

Max Range: 5.9117. Min Range: -5.9117

Max Act: 5.6305. Min Act: -0.1700

Data Index: 1389875 (Python Code)

Max Activating Token Index: 327

Text #5

Max Range: 5.9117. Min Range: -5.9117

Max Act: 5.6305. Min Act: -0.1700

Data Index: 1465306 (Python Code)

Max Activating Token Index: 48

Text #6

Max Range: 5.9117. Min Range: -5.9117

Max Act: 5.5992. Min Act: -0.1700

Data Index: 1525542 (Python Code)

Max Activating Token Index: 53

Text #7

Max Range: 5.9117. Min Range: -5.9117

Max Act: 5.5680. Min Act: -0.1700

Data Index: 1438525 (Python Code)

Max Activating Token Index: 64

Text #8

Max Range: 5.9117. Min Range: -5.9117

Max Act: 5.5992. Min Act: -0.1700

Data Index: 1633227 (Python Code)

Max Activating Token Index: 275

Text #9

Max Range: 5.9117. Min Range: -5.9117

Max Act: 5.5680. Min Act: -0.1700

Data Index: 1441450 (Python Code)

Max Activating Token Index: 267

Text #10

Max Range: 5.9117. Min Range: -5.9117

Max Act: 5.4742. Min Act: -0.1700

Data Index: 1535613 (Python Code)

Max Activating Token Index: 225

Text #11

Max Range: 5.9117. Min Range: -5.9117

Max Act: 5.5367. Min Act: -0.1700

Data Index: 1602631 (Python Code)

Max Activating Token Index: 194

Text #12

Max Range: 5.9117. Min Range: -5.9117

Max Act: 5.4430. Min Act: -0.1700

Data Index: 1597488 (Python Code)

Max Activating Token Index: 265

Text #13

Max Range: 5.9117. Min Range: -5.9117

Max Act: 5.5055. Min Act: -0.1700

Data Index: 1580067 (Python Code)

Max Activating Token Index: 176

Text #14

Max Range: 5.9117. Min Range: -5.9117

Max Act: 5.5680. Min Act: -0.1700

Data Index: 1480812 (Python Code)

Max Activating Token Index: 266

Text #15

Max Range: 5.9117. Min Range: -5.9117

Max Act: 5.4742. Min Act: -0.1700

Data Index: 1401380 (Python Code)

Max Activating Token Index: 129

Text #16

Max Range: 5.9117. Min Range: -5.9117

Max Act: 5.5367. Min Act: -0.1700

Data Index: 1448339 (Python Code)

Max Activating Token Index: 85

Text #17

Max Range: 5.9117. Min Range: -5.9117

Max Act: 5.5055. Min Act: -0.1700

Data Index: 1453392 (Python Code)

Max Activating Token Index: 399

Text #18

Max Range: 5.9117. Min Range: -5.9117

Max Act: 5.4430. Min Act: -0.1700

Data Index: 1537884 (Python Code)

Max Activating Token Index: 133

Text #19

Max Range: 5.9117. Min Range: -5.9117

Max Act: 5.4430. Min Act: -0.1700

Data Index: 1557905 (Python Code)

Max Activating Token Index: 764