Model: GELU Model: 3 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1290 in Layer 0

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('gelu-3l')

Text #0

Max Range: 3.5657. Min Range: -3.5657

Max Act: 3.5657. Min Act: -0.1700

Data Index: 779067 (C4 (Web Text))

Max Activating Token Index: 865

Text #1

Max Range: 3.5657. Min Range: -3.5657

Max Act: 3.5187. Min Act: -0.1700

Data Index: 536102 (C4 (Web Text))

Max Activating Token Index: 845

Text #2

Max Range: 3.5657. Min Range: -3.5657

Max Act: 3.4246. Min Act: -0.1700

Data Index: 585609 (C4 (Web Text))

Max Activating Token Index: 746

Text #3

Max Range: 3.5657. Min Range: -3.5657

Max Act: 3.4090. Min Act: -0.1700

Data Index: 298290 (C4 (Web Text))

Max Activating Token Index: 296

Text #4

Max Range: 3.5657. Min Range: -3.5657

Max Act: 3.4090. Min Act: -0.1700

Data Index: 911782 (C4 (Web Text))

Max Activating Token Index: 506

Text #5

Max Range: 3.5657. Min Range: -3.5657

Max Act: 3.3933. Min Act: -0.1700

Data Index: 610665 (C4 (Web Text))

Max Activating Token Index: 87

Text #6

Max Range: 3.5657. Min Range: -3.5657

Max Act: 3.3619. Min Act: -0.1700

Data Index: 219796 (C4 (Web Text))

Max Activating Token Index: 435

Text #7

Max Range: 3.5657. Min Range: -3.5657

Max Act: 3.3776. Min Act: -0.1700

Data Index: 386102 (C4 (Web Text))

Max Activating Token Index: 82

Text #8

Max Range: 3.5657. Min Range: -3.5657

Max Act: 3.3619. Min Act: -0.1700

Data Index: 1025169 (C4 (Web Text))

Max Activating Token Index: 283

Text #9

Max Range: 3.5657. Min Range: -3.5657

Max Act: 3.3305. Min Act: -0.1700

Data Index: 281093 (C4 (Web Text))

Max Activating Token Index: 774

Text #10

Max Range: 3.5657. Min Range: -3.5657

Max Act: 3.3305. Min Act: -0.1700

Data Index: 640116 (C4 (Web Text))

Max Activating Token Index: 93

Text #11

Max Range: 3.5657. Min Range: -3.5657

Max Act: 3.3305. Min Act: -0.1700

Data Index: 1293088 (C4 (Web Text))

Max Activating Token Index: 398

Text #12

Max Range: 3.5657. Min Range: -3.5657

Max Act: 3.3148. Min Act: -0.1700

Data Index: 860766 (C4 (Web Text))

Max Activating Token Index: 165

Text #13

Max Range: 3.5657. Min Range: -3.5657

Max Act: 3.3148. Min Act: -0.1700

Data Index: 128501 (C4 (Web Text))

Max Activating Token Index: 637

Text #14

Max Range: 3.5657. Min Range: -3.5657

Max Act: 3.2991. Min Act: -0.1700

Data Index: 1057590 (C4 (Web Text))

Max Activating Token Index: 810

Text #15

Max Range: 3.5657. Min Range: -3.5657

Max Act: 3.2834. Min Act: -0.1700

Data Index: 1322930 (C4 (Web Text))

Max Activating Token Index: 149

Text #16

Max Range: 3.5657. Min Range: -3.5657

Max Act: 3.2677. Min Act: -0.1700

Data Index: 858446 (C4 (Web Text))

Max Activating Token Index: 136

Text #17

Max Range: 3.5657. Min Range: -3.5657

Max Act: 3.2362. Min Act: -0.1700

Data Index: 3106 (C4 (Web Text))

Max Activating Token Index: 543

Text #18

Max Range: 3.5657. Min Range: -3.5657

Max Act: 3.2520. Min Act: -0.1700

Data Index: 817698 (C4 (Web Text))

Max Activating Token Index: 1004

Text #19

Max Range: 3.5657. Min Range: -3.5657

Max Act: 3.2520. Min Act: -0.1700

Data Index: 130698 (C4 (Web Text))

Max Activating Token Index: 664