Model: GELU Model: 4 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1229 in Layer 3

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('gelu-4l')

Text #0

Max Range: 7.7988. Min Range: -7.7988

Max Act: 7.7988. Min Act: -0.1700

Data Index: 1097334 (C4 (Web Text))

Max Activating Token Index: 166

Text #1

Max Range: 7.7988. Min Range: -7.7988

Max Act: 7.7676. Min Act: -0.1700

Data Index: 1572407 (Python Code)

Max Activating Token Index: 220

Text #2

Max Range: 7.7988. Min Range: -7.7988

Max Act: 7.7676. Min Act: -0.1700

Data Index: 87202 (C4 (Web Text))

Max Activating Token Index: 451

Text #3

Max Range: 7.7988. Min Range: -7.7988

Max Act: 7.6426. Min Act: -0.1700

Data Index: 1150279 (C4 (Web Text))

Max Activating Token Index: 745

Text #4

Max Range: 7.7988. Min Range: -7.7988

Max Act: 7.6738. Min Act: -0.1700

Data Index: 1483730 (Python Code)

Max Activating Token Index: 543

Text #5

Max Range: 7.7988. Min Range: -7.7988

Max Act: 7.7363. Min Act: -0.1700

Data Index: 1258393 (C4 (Web Text))

Max Activating Token Index: 654

Text #6

Max Range: 7.7988. Min Range: -7.7988

Max Act: 7.7051. Min Act: -0.1700

Data Index: 139595 (C4 (Web Text))

Max Activating Token Index: 749

Text #7

Max Range: 7.7988. Min Range: -7.7988

Max Act: 7.6738. Min Act: -0.1700

Data Index: 502691 (C4 (Web Text))

Max Activating Token Index: 265

Text #8

Max Range: 7.7988. Min Range: -7.7988

Max Act: 7.5801. Min Act: -0.1700

Data Index: 458028 (C4 (Web Text))

Max Activating Token Index: 695

Text #9

Max Range: 7.7988. Min Range: -7.7988

Max Act: 7.5488. Min Act: -0.1700

Data Index: 1091301 (C4 (Web Text))

Max Activating Token Index: 469

Text #10

Max Range: 7.7988. Min Range: -7.7988

Max Act: 7.4863. Min Act: -0.1700

Data Index: 1477172 (Python Code)

Max Activating Token Index: 228

Text #11

Max Range: 7.7988. Min Range: -7.7988

Max Act: 7.5488. Min Act: -0.1700

Data Index: 892007 (C4 (Web Text))

Max Activating Token Index: 584

Text #12

Max Range: 7.7988. Min Range: -7.7988

Max Act: 7.5176. Min Act: -0.1700

Data Index: 775612 (C4 (Web Text))

Max Activating Token Index: 211

Text #13

Max Range: 7.7988. Min Range: -7.7988

Max Act: 7.4238. Min Act: -0.1700

Data Index: 530681 (C4 (Web Text))

Max Activating Token Index: 517

Text #14

Max Range: 7.7988. Min Range: -7.7988

Max Act: 7.3926. Min Act: -0.1700

Data Index: 1333929 (C4 (Web Text))

Max Activating Token Index: 300

Text #15

Max Range: 7.7988. Min Range: -7.7988

Max Act: 7.3926. Min Act: -0.1700

Data Index: 563701 (C4 (Web Text))

Max Activating Token Index: 539

Text #16

Max Range: 7.7988. Min Range: -7.7988

Max Act: 7.3613. Min Act: -0.1700

Data Index: 1315004 (C4 (Web Text))

Max Activating Token Index: 297

Text #17

Max Range: 7.7988. Min Range: -7.7988

Max Act: 7.4238. Min Act: -0.1700

Data Index: 110561 (C4 (Web Text))

Max Activating Token Index: 505

Text #18

Max Range: 7.7988. Min Range: -7.7988

Max Act: 7.4551. Min Act: -0.1700

Data Index: 1273563 (C4 (Web Text))

Max Activating Token Index: 391

Text #19

Max Range: 7.7988. Min Range: -7.7988

Max Act: 7.3301. Min Act: -0.1700

Data Index: 1536113 (Python Code)

Max Activating Token Index: 281