Model: GELU Model: 2 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 308 in Layer 1

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('gelu-2l')

Text #0

Max Range: 5.8929. Min Range: -5.8929

Max Act: 5.8929. Min Act: -0.1700

Data Index: 329604 (C4 (Web Text))

Max Activating Token Index: 650

Text #1

Max Range: 5.8929. Min Range: -5.8929

Max Act: 5.8304. Min Act: -0.1700

Data Index: 32877 (C4 (Web Text))

Max Activating Token Index: 547

Text #2

Max Range: 5.8929. Min Range: -5.8929

Max Act: 5.7679. Min Act: -0.1700

Data Index: 1321463 (C4 (Web Text))

Max Activating Token Index: 46

Text #3

Max Range: 5.8929. Min Range: -5.8929

Max Act: 5.7679. Min Act: -0.1700

Data Index: 733001 (C4 (Web Text))

Max Activating Token Index: 453

Text #4

Max Range: 5.8929. Min Range: -5.8929

Max Act: 5.7054. Min Act: -0.1700

Data Index: 878948 (C4 (Web Text))

Max Activating Token Index: 585

Text #5

Max Range: 5.8929. Min Range: -5.8929

Max Act: 5.7366. Min Act: -0.1700

Data Index: 720898 (C4 (Web Text))

Max Activating Token Index: 287

Text #6

Max Range: 5.8929. Min Range: -5.8929

Max Act: 5.7679. Min Act: -0.1700

Data Index: 123471 (C4 (Web Text))

Max Activating Token Index: 745

Text #7

Max Range: 5.8929. Min Range: -5.8929

Max Act: 5.7366. Min Act: -0.1700

Data Index: 178043 (C4 (Web Text))

Max Activating Token Index: 6

Text #8

Max Range: 5.8929. Min Range: -5.8929

Max Act: 5.7366. Min Act: -0.1700

Data Index: 1235674 (C4 (Web Text))

Max Activating Token Index: 588

Text #9

Max Range: 5.8929. Min Range: -5.8929

Max Act: 5.6741. Min Act: -0.1700

Data Index: 368486 (C4 (Web Text))

Max Activating Token Index: 282

Text #10

Max Range: 5.8929. Min Range: -5.8929

Max Act: 5.7366. Min Act: -0.1700

Data Index: 45437 (C4 (Web Text))

Max Activating Token Index: 905

Text #11

Max Range: 5.8929. Min Range: -5.8929

Max Act: 5.6741. Min Act: -0.1700

Data Index: 1259003 (C4 (Web Text))

Max Activating Token Index: 421

Text #12

Max Range: 5.8929. Min Range: -5.8929

Max Act: 5.7054. Min Act: -0.1700

Data Index: 1118051 (C4 (Web Text))

Max Activating Token Index: 519

Text #13

Max Range: 5.8929. Min Range: -5.8929

Max Act: 5.7054. Min Act: -0.1700

Data Index: 1262829 (C4 (Web Text))

Max Activating Token Index: 226

Text #14

Max Range: 5.8929. Min Range: -5.8929

Max Act: 5.6429. Min Act: -0.1700

Data Index: 501783 (C4 (Web Text))

Max Activating Token Index: 462

Text #15

Max Range: 5.8929. Min Range: -5.8929

Max Act: 5.6741. Min Act: -0.1700

Data Index: 1007056 (C4 (Web Text))

Max Activating Token Index: 697

Text #16

Max Range: 5.8929. Min Range: -5.8929

Max Act: 5.6429. Min Act: -0.1700

Data Index: 67645 (C4 (Web Text))

Max Activating Token Index: 651

Text #17

Max Range: 5.8929. Min Range: -5.8929

Max Act: 5.6429. Min Act: -0.1700

Data Index: 485218 (C4 (Web Text))

Max Activating Token Index: 359

Text #18

Max Range: 5.8929. Min Range: -5.8929

Max Act: 5.6429. Min Act: -0.1700

Data Index: 1278788 (C4 (Web Text))

Max Activating Token Index: 254

Text #19

Max Range: 5.8929. Min Range: -5.8929

Max Act: 5.6741. Min Act: -0.1700

Data Index: 610367 (C4 (Web Text))

Max Activating Token Index: 606