Model: GELU Model: 4 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 667 in Layer 0

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('gelu-4l')

Text #0

Max Range: 3.0123. Min Range: -3.0123

Max Act: 2.9965. Min Act: -0.1700

Data Index: 907141 (C4 (Web Text))

Max Activating Token Index: 28

Text #1

Max Range: 3.0123. Min Range: -3.0123

Max Act: 3.0123. Min Act: -0.1700

Data Index: 658464 (C4 (Web Text))

Max Activating Token Index: 48

Text #2

Max Range: 3.0123. Min Range: -3.0123

Max Act: 2.9965. Min Act: -0.1700

Data Index: 1144998 (C4 (Web Text))

Max Activating Token Index: 551

Text #3

Max Range: 3.0123. Min Range: -3.0123

Max Act: 2.9807. Min Act: -0.1700

Data Index: 373075 (C4 (Web Text))

Max Activating Token Index: 185

Text #4

Max Range: 3.0123. Min Range: -3.0123

Max Act: 2.9965. Min Act: -0.1700

Data Index: 970525 (C4 (Web Text))

Max Activating Token Index: 250

Text #5

Max Range: 3.0123. Min Range: -3.0123

Max Act: 2.9807. Min Act: -0.1700

Data Index: 1089662 (C4 (Web Text))

Max Activating Token Index: 45

Text #6

Max Range: 3.0123. Min Range: -3.0123

Max Act: 2.9649. Min Act: -0.1700

Data Index: 1173303 (C4 (Web Text))

Max Activating Token Index: 148

Text #7

Max Range: 3.0123. Min Range: -3.0123

Max Act: 2.9807. Min Act: -0.1700

Data Index: 957830 (C4 (Web Text))

Max Activating Token Index: 88

Text #8

Max Range: 3.0123. Min Range: -3.0123

Max Act: 2.9490. Min Act: -0.1700

Data Index: 340557 (C4 (Web Text))

Max Activating Token Index: 44

Text #9

Max Range: 3.0123. Min Range: -3.0123

Max Act: 2.9649. Min Act: -0.1700

Data Index: 788266 (C4 (Web Text))

Max Activating Token Index: 1013

Text #10

Max Range: 3.0123. Min Range: -3.0123

Max Act: 2.9490. Min Act: -0.1700

Data Index: 1335363 (C4 (Web Text))

Max Activating Token Index: 337

Text #11

Max Range: 3.0123. Min Range: -3.0123

Max Act: 2.9490. Min Act: -0.1700

Data Index: 748843 (C4 (Web Text))

Max Activating Token Index: 213

Text #12

Max Range: 3.0123. Min Range: -3.0123

Max Act: 2.9332. Min Act: -0.1700

Data Index: 617580 (C4 (Web Text))

Max Activating Token Index: 152

Text #13

Max Range: 3.0123. Min Range: -3.0123

Max Act: 2.9173. Min Act: -0.1700

Data Index: 187151 (C4 (Web Text))

Max Activating Token Index: 484

Text #14

Max Range: 3.0123. Min Range: -3.0123

Max Act: 2.9332. Min Act: -0.1700

Data Index: 27298 (C4 (Web Text))

Max Activating Token Index: 202

Text #15

Max Range: 3.0123. Min Range: -3.0123

Max Act: 2.9332. Min Act: -0.1700

Data Index: 183222 (C4 (Web Text))

Max Activating Token Index: 760

Text #16

Max Range: 3.0123. Min Range: -3.0123

Max Act: 2.9173. Min Act: -0.1700

Data Index: 104191 (C4 (Web Text))

Max Activating Token Index: 252

Text #17

Max Range: 3.0123. Min Range: -3.0123

Max Act: 2.9015. Min Act: -0.1700

Data Index: 764417 (C4 (Web Text))

Max Activating Token Index: 626

Text #18

Max Range: 3.0123. Min Range: -3.0123

Max Act: 2.9173. Min Act: -0.1700

Data Index: 759651 (C4 (Web Text))

Max Activating Token Index: 288

Text #19

Max Range: 3.0123. Min Range: -3.0123

Max Act: 2.9015. Min Act: -0.1700

Data Index: 247304 (C4 (Web Text))

Max Activating Token Index: 76