Model: GELU Model: 2 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 562 in Layer 0

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('gelu-2l')

Text #0

Max Range: 3.2365. Min Range: -3.2365

Max Act: 3.2365. Min Act: -0.1700

Data Index: 947113 (C4 (Web Text))

Max Activating Token Index: 776

Text #1

Max Range: 3.2365. Min Range: -3.2365

Max Act: 3.1736. Min Act: -0.1700

Data Index: 1227174 (C4 (Web Text))

Max Activating Token Index: 561

Text #2

Max Range: 3.2365. Min Range: -3.2365

Max Act: 3.1421. Min Act: -0.1700

Data Index: 1265638 (C4 (Web Text))

Max Activating Token Index: 694

Text #3

Max Range: 3.2365. Min Range: -3.2365

Max Act: 3.1578. Min Act: -0.1700

Data Index: 169399 (C4 (Web Text))

Max Activating Token Index: 602

Text #4

Max Range: 3.2365. Min Range: -3.2365

Max Act: 3.1263. Min Act: -0.1700

Data Index: 354031 (C4 (Web Text))

Max Activating Token Index: 181

Text #5

Max Range: 3.2365. Min Range: -3.2365

Max Act: 3.1263. Min Act: -0.1700

Data Index: 452407 (C4 (Web Text))

Max Activating Token Index: 258

Text #6

Max Range: 3.2365. Min Range: -3.2365

Max Act: 3.1106. Min Act: -0.1700

Data Index: 1136724 (C4 (Web Text))

Max Activating Token Index: 126

Text #7

Max Range: 3.2365. Min Range: -3.2365

Max Act: 3.0948. Min Act: -0.1700

Data Index: 726457 (C4 (Web Text))

Max Activating Token Index: 1004

Text #8

Max Range: 3.2365. Min Range: -3.2365

Max Act: 3.0948. Min Act: -0.1700

Data Index: 61581 (C4 (Web Text))

Max Activating Token Index: 918

Text #9

Max Range: 3.2365. Min Range: -3.2365

Max Act: 3.1106. Min Act: -0.1700

Data Index: 307593 (C4 (Web Text))

Max Activating Token Index: 270

Text #10

Max Range: 3.2365. Min Range: -3.2365

Max Act: 3.1106. Min Act: -0.1700

Data Index: 477469 (C4 (Web Text))

Max Activating Token Index: 145

Text #11

Max Range: 3.2365. Min Range: -3.2365

Max Act: 3.0948. Min Act: -0.1700

Data Index: 300892 (C4 (Web Text))

Max Activating Token Index: 712

Text #12

Max Range: 3.2365. Min Range: -3.2365

Max Act: 3.0948. Min Act: -0.1700

Data Index: 1161217 (C4 (Web Text))

Max Activating Token Index: 134

Text #13

Max Range: 3.2365. Min Range: -3.2365

Max Act: 3.0790. Min Act: -0.1700

Data Index: 520795 (C4 (Web Text))

Max Activating Token Index: 215

Text #14

Max Range: 3.2365. Min Range: -3.2365

Max Act: 3.0632. Min Act: -0.1700

Data Index: 991757 (C4 (Web Text))

Max Activating Token Index: 488

Text #15

Max Range: 3.2365. Min Range: -3.2365

Max Act: 3.0632. Min Act: -0.1700

Data Index: 1121697 (C4 (Web Text))

Max Activating Token Index: 152

Text #16

Max Range: 3.2365. Min Range: -3.2365

Max Act: 3.0632. Min Act: -0.1700

Data Index: 1332862 (C4 (Web Text))

Max Activating Token Index: 8

Text #17

Max Range: 3.2365. Min Range: -3.2365

Max Act: 3.0632. Min Act: -0.1700

Data Index: 566086 (C4 (Web Text))

Max Activating Token Index: 625

Text #18

Max Range: 3.2365. Min Range: -3.2365

Max Act: 3.0632. Min Act: -0.1700

Data Index: 1047423 (C4 (Web Text))

Max Activating Token Index: 86

Text #19

Max Range: 3.2365. Min Range: -3.2365

Max Act: 3.0632. Min Act: -0.1700

Data Index: 1088966 (C4 (Web Text))

Max Activating Token Index: 321