Model: GELU Model: 4 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1526 in Layer 0

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('gelu-4l')

Text #0

Max Range: 3.1145. Min Range: -3.1145

Max Act: 3.1145. Min Act: -0.1700

Data Index: 672216 (C4 (Web Text))

Max Activating Token Index: 46

Text #1

Max Range: 3.1145. Min Range: -3.1145

Max Act: 3.0514. Min Act: -0.1700

Data Index: 1087769 (C4 (Web Text))

Max Activating Token Index: 301

Text #2

Max Range: 3.1145. Min Range: -3.1145

Max Act: 3.0040. Min Act: -0.1700

Data Index: 512591 (C4 (Web Text))

Max Activating Token Index: 1019

Text #3

Max Range: 3.1145. Min Range: -3.1145

Max Act: 2.9565. Min Act: -0.1700

Data Index: 1615747 (Python Code)

Max Activating Token Index: 8

Text #4

Max Range: 3.1145. Min Range: -3.1145

Max Act: 2.9249. Min Act: -0.1700

Data Index: 1188155 (C4 (Web Text))

Max Activating Token Index: 822

Text #5

Max Range: 3.1145. Min Range: -3.1145

Max Act: 2.9407. Min Act: -0.1700

Data Index: 768004 (C4 (Web Text))

Max Activating Token Index: 562

Text #6

Max Range: 3.1145. Min Range: -3.1145

Max Act: 2.9249. Min Act: -0.1700

Data Index: 1237067 (C4 (Web Text))

Max Activating Token Index: 391

Text #7

Max Range: 3.1145. Min Range: -3.1145

Max Act: 2.9249. Min Act: -0.1700

Data Index: 1514262 (Python Code)

Max Activating Token Index: 134

Text #8

Max Range: 3.1145. Min Range: -3.1145

Max Act: 2.9407. Min Act: -0.1700

Data Index: 1276250 (C4 (Web Text))

Max Activating Token Index: 991

Text #9

Max Range: 3.1145. Min Range: -3.1145

Max Act: 2.8931. Min Act: -0.1700

Data Index: 301658 (C4 (Web Text))

Max Activating Token Index: 989

Text #10

Max Range: 3.1145. Min Range: -3.1145

Max Act: 2.9249. Min Act: -0.1700

Data Index: 193656 (C4 (Web Text))

Max Activating Token Index: 590

Text #11

Max Range: 3.1145. Min Range: -3.1145

Max Act: 2.9249. Min Act: -0.1700

Data Index: 1331788 (C4 (Web Text))

Max Activating Token Index: 847

Text #12

Max Range: 3.1145. Min Range: -3.1145

Max Act: 2.9090. Min Act: -0.1700

Data Index: 606102 (C4 (Web Text))

Max Activating Token Index: 449

Text #13

Max Range: 3.1145. Min Range: -3.1145

Max Act: 2.8931. Min Act: -0.1700

Data Index: 472377 (C4 (Web Text))

Max Activating Token Index: 617

Text #14

Max Range: 3.1145. Min Range: -3.1145

Max Act: 2.8931. Min Act: -0.1700

Data Index: 1235589 (C4 (Web Text))

Max Activating Token Index: 67

Text #15

Max Range: 3.1145. Min Range: -3.1145

Max Act: 2.8614. Min Act: -0.1700

Data Index: 1386859 (Python Code)

Max Activating Token Index: 648

Text #16

Max Range: 3.1145. Min Range: -3.1145

Max Act: 2.8614. Min Act: -0.1700

Data Index: 1367161 (Python Code)

Max Activating Token Index: 927

Text #17

Max Range: 3.1145. Min Range: -3.1145

Max Act: 2.8931. Min Act: -0.1700

Data Index: 1192641 (C4 (Web Text))

Max Activating Token Index: 142

Text #18

Max Range: 3.1145. Min Range: -3.1145

Max Act: 2.8773. Min Act: -0.1700

Data Index: 916984 (C4 (Web Text))

Max Activating Token Index: 675

Text #19

Max Range: 3.1145. Min Range: -3.1145

Max Act: 2.8773. Min Act: -0.1700

Data Index: 1164910 (C4 (Web Text))

Max Activating Token Index: 401