Model: GELU Model: 4 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 972 in Layer 0

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('gelu-4l')

Text #0

Max Range: 3.7831. Min Range: -3.7831

Max Act: 3.7831. Min Act: -0.1700

Data Index: 787800 (C4 (Web Text))

Max Activating Token Index: 61

Text #1

Max Range: 3.7831. Min Range: -3.7831

Max Act: 3.7205. Min Act: -0.1700

Data Index: 664642 (C4 (Web Text))

Max Activating Token Index: 153

Text #2

Max Range: 3.7831. Min Range: -3.7831

Max Act: 3.6892. Min Act: -0.1700

Data Index: 289182 (C4 (Web Text))

Max Activating Token Index: 902

Text #3

Max Range: 3.7831. Min Range: -3.7831

Max Act: 3.6736. Min Act: -0.1699

Data Index: 126883 (C4 (Web Text))

Max Activating Token Index: 7

Text #4

Max Range: 3.7831. Min Range: -3.7831

Max Act: 3.5953. Min Act: -0.1700

Data Index: 380840 (C4 (Web Text))

Max Activating Token Index: 755

Text #5

Max Range: 3.7831. Min Range: -3.7831

Max Act: 3.5797. Min Act: -0.1700

Data Index: 666715 (C4 (Web Text))

Max Activating Token Index: 62

Text #6

Max Range: 3.7831. Min Range: -3.7831

Max Act: 3.5640. Min Act: -0.1700

Data Index: 1358426 (C4 (Web Text))

Max Activating Token Index: 111

Text #7

Max Range: 3.7831. Min Range: -3.7831

Max Act: 3.5640. Min Act: -0.1700

Data Index: 550683 (C4 (Web Text))

Max Activating Token Index: 340

Text #8

Max Range: 3.7831. Min Range: -3.7831

Max Act: 3.5483. Min Act: -0.1700

Data Index: 212045 (C4 (Web Text))

Max Activating Token Index: 6

Text #9

Max Range: 3.7831. Min Range: -3.7831

Max Act: 3.5327. Min Act: -0.1700

Data Index: 461813 (C4 (Web Text))

Max Activating Token Index: 1008

Text #10

Max Range: 3.7831. Min Range: -3.7831

Max Act: 3.5170. Min Act: -0.1700

Data Index: 78770 (C4 (Web Text))

Max Activating Token Index: 915

Text #11

Max Range: 3.7831. Min Range: -3.7831

Max Act: 3.5483. Min Act: -0.1700

Data Index: 420646 (C4 (Web Text))

Max Activating Token Index: 596

Text #12

Max Range: 3.7831. Min Range: -3.7831

Max Act: 3.5327. Min Act: -0.1700

Data Index: 1039925 (C4 (Web Text))

Max Activating Token Index: 270

Text #13

Max Range: 3.7831. Min Range: -3.7831

Max Act: 3.5013. Min Act: -0.1700

Data Index: 658736 (C4 (Web Text))

Max Activating Token Index: 85

Text #14

Max Range: 3.7831. Min Range: -3.7831

Max Act: 3.5327. Min Act: -0.1700

Data Index: 487826 (C4 (Web Text))

Max Activating Token Index: 40

Text #15

Max Range: 3.7831. Min Range: -3.7831

Max Act: 3.5170. Min Act: -0.1700

Data Index: 681999 (C4 (Web Text))

Max Activating Token Index: 412

Text #16

Max Range: 3.7831. Min Range: -3.7831

Max Act: 3.5170. Min Act: -0.1700

Data Index: 1072667 (C4 (Web Text))

Max Activating Token Index: 66

Text #17

Max Range: 3.7831. Min Range: -3.7831

Max Act: 3.5013. Min Act: -0.1700

Data Index: 718558 (C4 (Web Text))

Max Activating Token Index: 13

Text #18

Max Range: 3.7831. Min Range: -3.7831

Max Act: 3.4857. Min Act: -0.1700

Data Index: 89144 (C4 (Web Text))

Max Activating Token Index: 796

Text #19

Max Range: 3.7831. Min Range: -3.7831

Max Act: 3.4700. Min Act: -0.1700

Data Index: 799407 (C4 (Web Text))

Max Activating Token Index: 970