Model: GELU Model: 4 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1346 in Layer 3

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('gelu-4l')

Text #0

Max Range: 5.6460. Min Range: -5.6460

Max Act: 5.6147. Min Act: -0.1700

Data Index: 717879 (C4 (Web Text))

Max Activating Token Index: 816

Text #1

Max Range: 5.6460. Min Range: -5.6460

Max Act: 5.6460. Min Act: -0.1700

Data Index: 130353 (C4 (Web Text))

Max Activating Token Index: 388

Text #2

Max Range: 5.6460. Min Range: -5.6460

Max Act: 5.5522. Min Act: -0.1700

Data Index: 1172941 (C4 (Web Text))

Max Activating Token Index: 327

Text #3

Max Range: 5.6460. Min Range: -5.6460

Max Act: 5.6147. Min Act: -0.1700

Data Index: 87437 (C4 (Web Text))

Max Activating Token Index: 821

Text #4

Max Range: 5.6460. Min Range: -5.6460

Max Act: 5.5522. Min Act: -0.1700

Data Index: 1155972 (C4 (Web Text))

Max Activating Token Index: 821

Text #5

Max Range: 5.6460. Min Range: -5.6460

Max Act: 5.4272. Min Act: -0.1700

Data Index: 293794 (C4 (Web Text))

Max Activating Token Index: 326

Text #6

Max Range: 5.6460. Min Range: -5.6460

Max Act: 5.3335. Min Act: -0.1700

Data Index: 527096 (C4 (Web Text))

Max Activating Token Index: 947

Text #7

Max Range: 5.6460. Min Range: -5.6460

Max Act: 5.4272. Min Act: -0.1700

Data Index: 666646 (C4 (Web Text))

Max Activating Token Index: 875

Text #8

Max Range: 5.6460. Min Range: -5.6460

Max Act: 5.3647. Min Act: -0.1700

Data Index: 539063 (C4 (Web Text))

Max Activating Token Index: 450

Text #9

Max Range: 5.6460. Min Range: -5.6460

Max Act: 5.2710. Min Act: -0.1700

Data Index: 108085 (C4 (Web Text))

Max Activating Token Index: 507

Text #10

Max Range: 5.6460. Min Range: -5.6460

Max Act: 5.3335. Min Act: -0.1700

Data Index: 431433 (C4 (Web Text))

Max Activating Token Index: 426

Text #11

Max Range: 5.6460. Min Range: -5.6460

Max Act: 5.2710. Min Act: -0.1700

Data Index: 1261772 (C4 (Web Text))

Max Activating Token Index: 563

Text #12

Max Range: 5.6460. Min Range: -5.6460

Max Act: 5.2397. Min Act: -0.1700

Data Index: 324774 (C4 (Web Text))

Max Activating Token Index: 301

Text #13

Max Range: 5.6460. Min Range: -5.6460

Max Act: 5.1147. Min Act: -0.1700

Data Index: 1629784 (Python Code)

Max Activating Token Index: 901

Text #14

Max Range: 5.6460. Min Range: -5.6460

Max Act: 5.0835. Min Act: -0.1700

Data Index: 302342 (C4 (Web Text))

Max Activating Token Index: 807

Text #15

Max Range: 5.6460. Min Range: -5.6460

Max Act: 5.1147. Min Act: -0.1700

Data Index: 1161242 (C4 (Web Text))

Max Activating Token Index: 855

Text #16

Max Range: 5.6460. Min Range: -5.6460

Max Act: 5.1147. Min Act: -0.1700

Data Index: 1195439 (C4 (Web Text))

Max Activating Token Index: 354

Text #17

Max Range: 5.6460. Min Range: -5.6460

Max Act: 5.0835. Min Act: -0.1700

Data Index: 924438 (C4 (Web Text))

Max Activating Token Index: 224

Text #18

Max Range: 5.6460. Min Range: -5.6460

Max Act: 4.9897. Min Act: -0.1700

Data Index: 1039718 (C4 (Web Text))

Max Activating Token Index: 141

Text #19

Max Range: 5.6460. Min Range: -5.6460

Max Act: 4.9897. Min Act: -0.1700

Data Index: 733741 (C4 (Web Text))

Max Activating Token Index: 963