Model: GELU Model: 3 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 599 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('gelu-3l')

Text #0

Max Range: 6.1302. Min Range: -6.1302

Max Act: 6.1302. Min Act: -0.1700

Data Index: 1190539 (C4 (Web Text))

Max Activating Token Index: 919

Text #1

Max Range: 6.1302. Min Range: -6.1302

Max Act: 5.9740. Min Act: -0.1700

Data Index: 92995 (C4 (Web Text))

Max Activating Token Index: 155

Text #2

Max Range: 6.1302. Min Range: -6.1302

Max Act: 5.9115. Min Act: -0.1700

Data Index: 832277 (C4 (Web Text))

Max Activating Token Index: 755

Text #3

Max Range: 6.1302. Min Range: -6.1302

Max Act: 5.9427. Min Act: -0.1700

Data Index: 584917 (C4 (Web Text))

Max Activating Token Index: 382

Text #4

Max Range: 6.1302. Min Range: -6.1302

Max Act: 5.7865. Min Act: -0.1700

Data Index: 741409 (C4 (Web Text))

Max Activating Token Index: 41

Text #5

Max Range: 6.1302. Min Range: -6.1302

Max Act: 5.6927. Min Act: -0.1700

Data Index: 1300642 (C4 (Web Text))

Max Activating Token Index: 468

Text #6

Max Range: 6.1302. Min Range: -6.1302

Max Act: 5.7552. Min Act: -0.1700

Data Index: 934147 (C4 (Web Text))

Max Activating Token Index: 214

Text #7

Max Range: 6.1302. Min Range: -6.1302

Max Act: 5.7240. Min Act: -0.1700

Data Index: 254331 (C4 (Web Text))

Max Activating Token Index: 209

Text #8

Max Range: 6.1302. Min Range: -6.1302

Max Act: 5.7240. Min Act: -0.1700

Data Index: 613722 (C4 (Web Text))

Max Activating Token Index: 394

Text #9

Max Range: 6.1302. Min Range: -6.1302

Max Act: 5.7240. Min Act: -0.1700

Data Index: 840392 (C4 (Web Text))

Max Activating Token Index: 427

Text #10

Max Range: 6.1302. Min Range: -6.1302

Max Act: 5.6302. Min Act: -0.1700

Data Index: 1310760 (C4 (Web Text))

Max Activating Token Index: 422

Text #11

Max Range: 6.1302. Min Range: -6.1302

Max Act: 5.5677. Min Act: -0.1700

Data Index: 1357705 (C4 (Web Text))

Max Activating Token Index: 330

Text #12

Max Range: 6.1302. Min Range: -6.1302

Max Act: 5.6615. Min Act: -0.1700

Data Index: 156481 (C4 (Web Text))

Max Activating Token Index: 648

Text #13

Max Range: 6.1302. Min Range: -6.1302

Max Act: 5.6615. Min Act: -0.1700

Data Index: 807569 (C4 (Web Text))

Max Activating Token Index: 389

Text #14

Max Range: 6.1302. Min Range: -6.1302

Max Act: 5.5365. Min Act: -0.1700

Data Index: 774716 (C4 (Web Text))

Max Activating Token Index: 310

Text #15

Max Range: 6.1302. Min Range: -6.1302

Max Act: 5.5990. Min Act: -0.1700

Data Index: 443316 (C4 (Web Text))

Max Activating Token Index: 351

Text #16

Max Range: 6.1302. Min Range: -6.1302

Max Act: 5.5365. Min Act: -0.1700

Data Index: 996797 (C4 (Web Text))

Max Activating Token Index: 77

Text #17

Max Range: 6.1302. Min Range: -6.1302

Max Act: 5.4115. Min Act: -0.1700

Data Index: 690001 (C4 (Web Text))

Max Activating Token Index: 455

Text #18

Max Range: 6.1302. Min Range: -6.1302

Max Act: 5.5052. Min Act: -0.1700

Data Index: 930384 (C4 (Web Text))

Max Activating Token Index: 287

Text #19

Max Range: 6.1302. Min Range: -6.1302

Max Act: 5.5052. Min Act: -0.1700

Data Index: 1172951 (C4 (Web Text))

Max Activating Token Index: 382