Model: GELU Model: 3 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1346 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('gelu-3l')

Text #0

Max Range: 5.3463. Min Range: -5.3463

Max Act: 5.3463. Min Act: -0.1699

Data Index: 308333 (C4 (Web Text))

Max Activating Token Index: 593

Text #1

Max Range: 5.3463. Min Range: -5.3463

Max Act: 5.2526. Min Act: -0.1699

Data Index: 792891 (C4 (Web Text))

Max Activating Token Index: 219

Text #2

Max Range: 5.3463. Min Range: -5.3463

Max Act: 5.2526. Min Act: -0.1700

Data Index: 260423 (C4 (Web Text))

Max Activating Token Index: 832

Text #3

Max Range: 5.3463. Min Range: -5.3463

Max Act: 5.1588. Min Act: -0.1699

Data Index: 51854 (C4 (Web Text))

Max Activating Token Index: 692

Text #4

Max Range: 5.3463. Min Range: -5.3463

Max Act: 5.0963. Min Act: -0.1699

Data Index: 1096296 (C4 (Web Text))

Max Activating Token Index: 280

Text #5

Max Range: 5.3463. Min Range: -5.3463

Max Act: 5.0963. Min Act: -0.1700

Data Index: 1075288 (C4 (Web Text))

Max Activating Token Index: 927

Text #6

Max Range: 5.3463. Min Range: -5.3463

Max Act: 5.0963. Min Act: -0.1700

Data Index: 503647 (C4 (Web Text))

Max Activating Token Index: 891

Text #7

Max Range: 5.3463. Min Range: -5.3463

Max Act: 5.0338. Min Act: -0.1700

Data Index: 1079659 (C4 (Web Text))

Max Activating Token Index: 308

Text #8

Max Range: 5.3463. Min Range: -5.3463

Max Act: 5.0338. Min Act: -0.1700

Data Index: 783201 (C4 (Web Text))

Max Activating Token Index: 191

Text #9

Max Range: 5.3463. Min Range: -5.3463

Max Act: 5.0651. Min Act: -0.1700

Data Index: 530338 (C4 (Web Text))

Max Activating Token Index: 402

Text #10

Max Range: 5.3463. Min Range: -5.3463

Max Act: 5.0338. Min Act: -0.1700

Data Index: 1138241 (C4 (Web Text))

Max Activating Token Index: 513

Text #11

Max Range: 5.3463. Min Range: -5.3463

Max Act: 5.0651. Min Act: -0.1700

Data Index: 771235 (C4 (Web Text))

Max Activating Token Index: 92

Text #12

Max Range: 5.3463. Min Range: -5.3463

Max Act: 5.0338. Min Act: -0.1699

Data Index: 744014 (C4 (Web Text))

Max Activating Token Index: 885

Text #13

Max Range: 5.3463. Min Range: -5.3463

Max Act: 5.0026. Min Act: -0.1700

Data Index: 149062 (C4 (Web Text))

Max Activating Token Index: 462

Text #14

Max Range: 5.3463. Min Range: -5.3463

Max Act: 4.9401. Min Act: -0.1700

Data Index: 234385 (C4 (Web Text))

Max Activating Token Index: 960

Text #15

Max Range: 5.3463. Min Range: -5.3463

Max Act: 4.9401. Min Act: -0.1692

Data Index: 450059 (C4 (Web Text))

Max Activating Token Index: 351

Text #16

Max Range: 5.3463. Min Range: -5.3463

Max Act: 4.9088. Min Act: -0.1700

Data Index: 315654 (C4 (Web Text))

Max Activating Token Index: 682

Text #17

Max Range: 5.3463. Min Range: -5.3463

Max Act: 4.9401. Min Act: -0.1700

Data Index: 988629 (C4 (Web Text))

Max Activating Token Index: 893

Text #18

Max Range: 5.3463. Min Range: -5.3463

Max Act: 4.9401. Min Act: -0.1700

Data Index: 45801 (C4 (Web Text))

Max Activating Token Index: 463

Text #19

Max Range: 5.3463. Min Range: -5.3463

Max Act: 4.9401. Min Act: -0.1700

Data Index: 946547 (C4 (Web Text))

Max Activating Token Index: 631