Model: GELU Model: 4 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 531 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('gelu-4l')

Text #0

Max Range: 5.8676. Min Range: -5.8676

Max Act: 5.8363. Min Act: -0.1700

Data Index: 772574 (C4 (Web Text))

Max Activating Token Index: 1003

Text #1

Max Range: 5.8676. Min Range: -5.8676

Max Act: 5.8676. Min Act: -0.1700

Data Index: 757133 (C4 (Web Text))

Max Activating Token Index: 777

Text #2

Max Range: 5.8676. Min Range: -5.8676

Max Act: 5.8051. Min Act: -0.1700

Data Index: 1094949 (C4 (Web Text))

Max Activating Token Index: 788

Text #3

Max Range: 5.8676. Min Range: -5.8676

Max Act: 5.7426. Min Act: -0.1700

Data Index: 968973 (C4 (Web Text))

Max Activating Token Index: 816

Text #4

Max Range: 5.8676. Min Range: -5.8676

Max Act: 5.7426. Min Act: -0.1700

Data Index: 1270177 (C4 (Web Text))

Max Activating Token Index: 897

Text #5

Max Range: 5.8676. Min Range: -5.8676

Max Act: 5.7113. Min Act: -0.1700

Data Index: 969313 (C4 (Web Text))

Max Activating Token Index: 905

Text #6

Max Range: 5.8676. Min Range: -5.8676

Max Act: 5.7113. Min Act: -0.1700

Data Index: 1194510 (C4 (Web Text))

Max Activating Token Index: 290

Text #7

Max Range: 5.8676. Min Range: -5.8676

Max Act: 5.6488. Min Act: -0.1700

Data Index: 1167536 (C4 (Web Text))

Max Activating Token Index: 889

Text #8

Max Range: 5.8676. Min Range: -5.8676

Max Act: 5.6488. Min Act: -0.1700

Data Index: 112775 (C4 (Web Text))

Max Activating Token Index: 716

Text #9

Max Range: 5.8676. Min Range: -5.8676

Max Act: 5.5551. Min Act: -0.1700

Data Index: 1182404 (C4 (Web Text))

Max Activating Token Index: 459

Text #10

Max Range: 5.8676. Min Range: -5.8676

Max Act: 5.5238. Min Act: -0.1700

Data Index: 1030400 (C4 (Web Text))

Max Activating Token Index: 389

Text #11

Max Range: 5.8676. Min Range: -5.8676

Max Act: 5.4926. Min Act: -0.1700

Data Index: 1062813 (C4 (Web Text))

Max Activating Token Index: 906

Text #12

Max Range: 5.8676. Min Range: -5.8676

Max Act: 5.4926. Min Act: -0.1700

Data Index: 1203575 (C4 (Web Text))

Max Activating Token Index: 746

Text #13

Max Range: 5.8676. Min Range: -5.8676

Max Act: 5.4301. Min Act: -0.1700

Data Index: 747158 (C4 (Web Text))

Max Activating Token Index: 662

Text #14

Max Range: 5.8676. Min Range: -5.8676

Max Act: 5.3988. Min Act: -0.1700

Data Index: 664883 (C4 (Web Text))

Max Activating Token Index: 902

Text #15

Max Range: 5.8676. Min Range: -5.8676

Max Act: 5.3363. Min Act: -0.1700

Data Index: 27538 (C4 (Web Text))

Max Activating Token Index: 842

Text #16

Max Range: 5.8676. Min Range: -5.8676

Max Act: 5.3051. Min Act: -0.1700

Data Index: 1029799 (C4 (Web Text))

Max Activating Token Index: 348

Text #17

Max Range: 5.8676. Min Range: -5.8676

Max Act: 5.3051. Min Act: -0.1700

Data Index: 1209075 (C4 (Web Text))

Max Activating Token Index: 405

Text #18

Max Range: 5.8676. Min Range: -5.8676

Max Act: 5.2738. Min Act: -0.1700

Data Index: 24308 (C4 (Web Text))

Max Activating Token Index: 430

Text #19

Max Range: 5.8676. Min Range: -5.8676

Max Act: 5.3051. Min Act: -0.1700

Data Index: 334080 (C4 (Web Text))

Max Activating Token Index: 687