Model: GELU Model: 3 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1280 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('gelu-3l')

Text #0

Max Range: 5.1783. Min Range: -5.1783

Max Act: 5.1783. Min Act: -0.1700

Data Index: 1198736 (C4 (Web Text))

Max Activating Token Index: 41

Text #1

Max Range: 5.1783. Min Range: -5.1783

Max Act: 5.1471. Min Act: -0.1700

Data Index: 877452 (C4 (Web Text))

Max Activating Token Index: 139

Text #2

Max Range: 5.1783. Min Range: -5.1783

Max Act: 5.1783. Min Act: -0.1700

Data Index: 411743 (C4 (Web Text))

Max Activating Token Index: 718

Text #3

Max Range: 5.1783. Min Range: -5.1783

Max Act: 5.1158. Min Act: -0.1700

Data Index: 280864 (C4 (Web Text))

Max Activating Token Index: 313

Text #4

Max Range: 5.1783. Min Range: -5.1783

Max Act: 5.0846. Min Act: -0.1700

Data Index: 777441 (C4 (Web Text))

Max Activating Token Index: 177

Text #5

Max Range: 5.1783. Min Range: -5.1783

Max Act: 5.0846. Min Act: -0.1700

Data Index: 372201 (C4 (Web Text))

Max Activating Token Index: 960

Text #6

Max Range: 5.1783. Min Range: -5.1783

Max Act: 4.9908. Min Act: -0.1700

Data Index: 253089 (C4 (Web Text))

Max Activating Token Index: 251

Text #7

Max Range: 5.1783. Min Range: -5.1783

Max Act: 4.9596. Min Act: -0.1700

Data Index: 311455 (C4 (Web Text))

Max Activating Token Index: 703

Text #8

Max Range: 5.1783. Min Range: -5.1783

Max Act: 4.9283. Min Act: -0.1700

Data Index: 427279 (C4 (Web Text))

Max Activating Token Index: 140

Text #9

Max Range: 5.1783. Min Range: -5.1783

Max Act: 4.9283. Min Act: -0.1700

Data Index: 241363 (C4 (Web Text))

Max Activating Token Index: 917

Text #10

Max Range: 5.1783. Min Range: -5.1783

Max Act: 4.9596. Min Act: -0.1700

Data Index: 314506 (C4 (Web Text))

Max Activating Token Index: 885

Text #11

Max Range: 5.1783. Min Range: -5.1783

Max Act: 4.9283. Min Act: -0.1700

Data Index: 507636 (C4 (Web Text))

Max Activating Token Index: 996

Text #12

Max Range: 5.1783. Min Range: -5.1783

Max Act: 4.8970. Min Act: -0.1700

Data Index: 1151811 (C4 (Web Text))

Max Activating Token Index: 456

Text #13

Max Range: 5.1783. Min Range: -5.1783

Max Act: 4.8970. Min Act: -0.1700

Data Index: 1087517 (C4 (Web Text))

Max Activating Token Index: 101

Text #14

Max Range: 5.1783. Min Range: -5.1783

Max Act: 4.9283. Min Act: -0.1700

Data Index: 172706 (C4 (Web Text))

Max Activating Token Index: 437

Text #15

Max Range: 5.1783. Min Range: -5.1783

Max Act: 4.8658. Min Act: -0.1699

Data Index: 571966 (C4 (Web Text))

Max Activating Token Index: 294

Text #16

Max Range: 5.1783. Min Range: -5.1783

Max Act: 4.8345. Min Act: -0.1700

Data Index: 751653 (C4 (Web Text))

Max Activating Token Index: 173

Text #17

Max Range: 5.1783. Min Range: -5.1783

Max Act: 4.8033. Min Act: -0.1700

Data Index: 434189 (C4 (Web Text))

Max Activating Token Index: 258

Text #18

Max Range: 5.1783. Min Range: -5.1783

Max Act: 4.8345. Min Act: -0.1700

Data Index: 656552 (C4 (Web Text))

Max Activating Token Index: 447

Text #19

Max Range: 5.1783. Min Range: -5.1783

Max Act: 4.8658. Min Act: -0.1700

Data Index: 313294 (C4 (Web Text))

Max Activating Token Index: 521