Model: GELU Model: 2 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 669 in Layer 1

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('gelu-2l')

Text #0

Max Range: 3.6494. Min Range: -3.6494

Max Act: 3.6494. Min Act: -0.1700

Data Index: 745874 (C4 (Web Text))

Max Activating Token Index: 776

Text #1

Max Range: 3.6494. Min Range: -3.6494

Max Act: 3.5554. Min Act: -0.1700

Data Index: 863529 (C4 (Web Text))

Max Activating Token Index: 39

Text #2

Max Range: 3.6494. Min Range: -3.6494

Max Act: 3.5241. Min Act: -0.1700

Data Index: 788484 (C4 (Web Text))

Max Activating Token Index: 652

Text #3

Max Range: 3.6494. Min Range: -3.6494

Max Act: 3.5084. Min Act: -0.1700

Data Index: 1209536 (C4 (Web Text))

Max Activating Token Index: 344

Text #4

Max Range: 3.6494. Min Range: -3.6494

Max Act: 3.5084. Min Act: -0.1700

Data Index: 712824 (C4 (Web Text))

Max Activating Token Index: 730

Text #5

Max Range: 3.6494. Min Range: -3.6494

Max Act: 3.4771. Min Act: -0.1700

Data Index: 190940 (C4 (Web Text))

Max Activating Token Index: 721

Text #6

Max Range: 3.6494. Min Range: -3.6494

Max Act: 3.4458. Min Act: -0.1700

Data Index: 319010 (C4 (Web Text))

Max Activating Token Index: 730

Text #7

Max Range: 3.6494. Min Range: -3.6494

Max Act: 3.4458. Min Act: -0.1700

Data Index: 1192203 (C4 (Web Text))

Max Activating Token Index: 542

Text #8

Max Range: 3.6494. Min Range: -3.6494

Max Act: 3.4458. Min Act: -0.1700

Data Index: 1215845 (C4 (Web Text))

Max Activating Token Index: 716

Text #9

Max Range: 3.6494. Min Range: -3.6494

Max Act: 3.4458. Min Act: -0.1700

Data Index: 1208656 (C4 (Web Text))

Max Activating Token Index: 900

Text #10

Max Range: 3.6494. Min Range: -3.6494

Max Act: 3.4301. Min Act: -0.1700

Data Index: 964541 (C4 (Web Text))

Max Activating Token Index: 809

Text #11

Max Range: 3.6494. Min Range: -3.6494

Max Act: 3.4301. Min Act: -0.1700

Data Index: 492238 (C4 (Web Text))

Max Activating Token Index: 196

Text #12

Max Range: 3.6494. Min Range: -3.6494

Max Act: 3.4458. Min Act: -0.1700

Data Index: 1238640 (C4 (Web Text))

Max Activating Token Index: 689

Text #13

Max Range: 3.6494. Min Range: -3.6494

Max Act: 3.4144. Min Act: -0.1700

Data Index: 557739 (C4 (Web Text))

Max Activating Token Index: 1021

Text #14

Max Range: 3.6494. Min Range: -3.6494

Max Act: 3.3673. Min Act: -0.1700

Data Index: 356918 (C4 (Web Text))

Max Activating Token Index: 523

Text #15

Max Range: 3.6494. Min Range: -3.6494

Max Act: 3.4144. Min Act: -0.1700

Data Index: 1118683 (C4 (Web Text))

Max Activating Token Index: 45

Text #16

Max Range: 3.6494. Min Range: -3.6494

Max Act: 3.4144. Min Act: -0.1700

Data Index: 1087022 (C4 (Web Text))

Max Activating Token Index: 604

Text #17

Max Range: 3.6494. Min Range: -3.6494

Max Act: 3.3673. Min Act: -0.1700

Data Index: 367448 (C4 (Web Text))

Max Activating Token Index: 332

Text #18

Max Range: 3.6494. Min Range: -3.6494

Max Act: 3.3987. Min Act: -0.1700

Data Index: 30424 (C4 (Web Text))

Max Activating Token Index: 586

Text #19

Max Range: 3.6494. Min Range: -3.6494

Max Act: 3.4144. Min Act: -0.1700

Data Index: 618397 (C4 (Web Text))

Max Activating Token Index: 491