Model: GELU Model: 4 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1048 in Layer 3

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('gelu-4l')

Text #0

Max Range: 5.3006. Min Range: -5.3006

Max Act: 5.3006. Min Act: -0.1700

Data Index: 1123509 (C4 (Web Text))

Max Activating Token Index: 947

Text #1

Max Range: 5.3006. Min Range: -5.3006

Max Act: 5.1444. Min Act: -0.1700

Data Index: 1113058 (C4 (Web Text))

Max Activating Token Index: 760

Text #2

Max Range: 5.3006. Min Range: -5.3006

Max Act: 5.0975. Min Act: -0.1700

Data Index: 1321521 (C4 (Web Text))

Max Activating Token Index: 811

Text #3

Max Range: 5.3006. Min Range: -5.3006

Max Act: 5.0819. Min Act: -0.1700

Data Index: 895162 (C4 (Web Text))

Max Activating Token Index: 422

Text #4

Max Range: 5.3006. Min Range: -5.3006

Max Act: 5.0506. Min Act: -0.1700

Data Index: 437024 (C4 (Web Text))

Max Activating Token Index: 253

Text #5

Max Range: 5.3006. Min Range: -5.3006

Max Act: 4.9725. Min Act: -0.1700

Data Index: 766662 (C4 (Web Text))

Max Activating Token Index: 352

Text #6

Max Range: 5.3006. Min Range: -5.3006

Max Act: 4.8944. Min Act: -0.1700

Data Index: 363461 (C4 (Web Text))

Max Activating Token Index: 208

Text #7

Max Range: 5.3006. Min Range: -5.3006

Max Act: 4.8631. Min Act: -0.1700

Data Index: 1615005 (Python Code)

Max Activating Token Index: 481

Text #8

Max Range: 5.3006. Min Range: -5.3006

Max Act: 4.9100. Min Act: -0.1700

Data Index: 328955 (C4 (Web Text))

Max Activating Token Index: 275

Text #9

Max Range: 5.3006. Min Range: -5.3006

Max Act: 4.9569. Min Act: -0.1700

Data Index: 337900 (C4 (Web Text))

Max Activating Token Index: 883

Text #10

Max Range: 5.3006. Min Range: -5.3006

Max Act: 4.9256. Min Act: -0.1700

Data Index: 1423343 (Python Code)

Max Activating Token Index: 628

Text #11

Max Range: 5.3006. Min Range: -5.3006

Max Act: 4.8475. Min Act: -0.1700

Data Index: 57041 (C4 (Web Text))

Max Activating Token Index: 941

Text #12

Max Range: 5.3006. Min Range: -5.3006

Max Act: 4.8631. Min Act: -0.1700

Data Index: 466106 (C4 (Web Text))

Max Activating Token Index: 469

Text #13

Max Range: 5.3006. Min Range: -5.3006

Max Act: 4.8475. Min Act: -0.1700

Data Index: 1098739 (C4 (Web Text))

Max Activating Token Index: 499

Text #14

Max Range: 5.3006. Min Range: -5.3006

Max Act: 4.8631. Min Act: -0.1700

Data Index: 231454 (C4 (Web Text))

Max Activating Token Index: 998

Text #15

Max Range: 5.3006. Min Range: -5.3006

Max Act: 4.8162. Min Act: -0.1699

Data Index: 207779 (C4 (Web Text))

Max Activating Token Index: 834

Text #16

Max Range: 5.3006. Min Range: -5.3006

Max Act: 4.7694. Min Act: -0.1700

Data Index: 1396633 (Python Code)

Max Activating Token Index: 985

Text #17

Max Range: 5.3006. Min Range: -5.3006

Max Act: 4.8006. Min Act: -0.1700

Data Index: 156753 (C4 (Web Text))

Max Activating Token Index: 716

Text #18

Max Range: 5.3006. Min Range: -5.3006

Max Act: 4.8006. Min Act: -0.1700

Data Index: 206115 (C4 (Web Text))

Max Activating Token Index: 610

Text #19

Max Range: 5.3006. Min Range: -5.3006

Max Act: 4.7850. Min Act: -0.1700

Data Index: 410429 (C4 (Web Text))

Max Activating Token Index: 971