Model: GELU Model: 4 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1559 in Layer 3

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('gelu-4l')

Text #0

Max Range: 5.1275. Min Range: -5.1275

Max Act: 5.1275. Min Act: -0.1700

Data Index: 920884 (C4 (Web Text))

Max Activating Token Index: 689

Text #1

Max Range: 5.1275. Min Range: -5.1275

Max Act: 4.9713. Min Act: -0.1700

Data Index: 468680 (C4 (Web Text))

Max Activating Token Index: 396

Text #2

Max Range: 5.1275. Min Range: -5.1275

Max Act: 4.8150. Min Act: -0.1700

Data Index: 199500 (C4 (Web Text))

Max Activating Token Index: 419

Text #3

Max Range: 5.1275. Min Range: -5.1275

Max Act: 4.7838. Min Act: -0.1700

Data Index: 1299674 (C4 (Web Text))

Max Activating Token Index: 434

Text #4

Max Range: 5.1275. Min Range: -5.1275

Max Act: 4.7838. Min Act: -0.1700

Data Index: 1311890 (C4 (Web Text))

Max Activating Token Index: 216

Text #5

Max Range: 5.1275. Min Range: -5.1275

Max Act: 4.7213. Min Act: -0.1700

Data Index: 1124117 (C4 (Web Text))

Max Activating Token Index: 101

Text #6

Max Range: 5.1275. Min Range: -5.1275

Max Act: 4.6588. Min Act: -0.1700

Data Index: 734607 (C4 (Web Text))

Max Activating Token Index: 83

Text #7

Max Range: 5.1275. Min Range: -5.1275

Max Act: 4.6275. Min Act: -0.1700

Data Index: 533844 (C4 (Web Text))

Max Activating Token Index: 186

Text #8

Max Range: 5.1275. Min Range: -5.1275

Max Act: 4.6588. Min Act: -0.1700

Data Index: 666950 (C4 (Web Text))

Max Activating Token Index: 689

Text #9

Max Range: 5.1275. Min Range: -5.1275

Max Act: 4.5963. Min Act: -0.1700

Data Index: 859755 (C4 (Web Text))

Max Activating Token Index: 188

Text #10

Max Range: 5.1275. Min Range: -5.1275

Max Act: 4.4869. Min Act: -0.1700

Data Index: 744936 (C4 (Web Text))

Max Activating Token Index: 419

Text #11

Max Range: 5.1275. Min Range: -5.1275

Max Act: 4.4400. Min Act: -0.1700

Data Index: 1151738 (C4 (Web Text))

Max Activating Token Index: 266

Text #12

Max Range: 5.1275. Min Range: -5.1275

Max Act: 4.3619. Min Act: -0.1700

Data Index: 385728 (C4 (Web Text))

Max Activating Token Index: 223

Text #13

Max Range: 5.1275. Min Range: -5.1275

Max Act: 4.4400. Min Act: -0.1700

Data Index: 1355656 (C4 (Web Text))

Max Activating Token Index: 118

Text #14

Max Range: 5.1275. Min Range: -5.1275

Max Act: 4.4712. Min Act: -0.1700

Data Index: 708175 (C4 (Web Text))

Max Activating Token Index: 226

Text #15

Max Range: 5.1275. Min Range: -5.1275

Max Act: 4.3775. Min Act: -0.1700

Data Index: 506120 (C4 (Web Text))

Max Activating Token Index: 56

Text #16

Max Range: 5.1275. Min Range: -5.1275

Max Act: 4.3931. Min Act: -0.1700

Data Index: 550041 (C4 (Web Text))

Max Activating Token Index: 465

Text #17

Max Range: 5.1275. Min Range: -5.1275

Max Act: 4.3775. Min Act: -0.1700

Data Index: 91877 (C4 (Web Text))

Max Activating Token Index: 164

Text #18

Max Range: 5.1275. Min Range: -5.1275

Max Act: 4.3931. Min Act: -0.1699

Data Index: 345366 (C4 (Web Text))

Max Activating Token Index: 618

Text #19

Max Range: 5.1275. Min Range: -5.1275

Max Act: 4.3775. Min Act: -0.1700

Data Index: 289321 (C4 (Web Text))

Max Activating Token Index: 915