Model: GELU Model: 4 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1935 in Layer 1

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('gelu-4l')

Text #0

Max Range: 3.3879. Min Range: -3.3879

Max Act: 3.3879. Min Act: -0.1700

Data Index: 938638 (C4 (Web Text))

Max Activating Token Index: 525

Text #1

Max Range: 3.3879. Min Range: -3.3879

Max Act: 3.1836. Min Act: -0.1700

Data Index: 329288 (C4 (Web Text))

Max Activating Token Index: 900

Text #2

Max Range: 3.3879. Min Range: -3.3879

Max Act: 3.1521. Min Act: -0.1700

Data Index: 613820 (C4 (Web Text))

Max Activating Token Index: 869

Text #3

Max Range: 3.3879. Min Range: -3.3879

Max Act: 3.1679. Min Act: -0.1700

Data Index: 205919 (C4 (Web Text))

Max Activating Token Index: 725

Text #4

Max Range: 3.3879. Min Range: -3.3879

Max Act: 3.1521. Min Act: -0.1700

Data Index: 96420 (C4 (Web Text))

Max Activating Token Index: 913

Text #5

Max Range: 3.3879. Min Range: -3.3879

Max Act: 3.1521. Min Act: -0.1700

Data Index: 1223591 (C4 (Web Text))

Max Activating Token Index: 972

Text #6

Max Range: 3.3879. Min Range: -3.3879

Max Act: 3.1049. Min Act: -0.1700

Data Index: 191241 (C4 (Web Text))

Max Activating Token Index: 969

Text #7

Max Range: 3.3879. Min Range: -3.3879

Max Act: 3.1206. Min Act: -0.1700

Data Index: 1010002 (C4 (Web Text))

Max Activating Token Index: 820

Text #8

Max Range: 3.3879. Min Range: -3.3879

Max Act: 3.1049. Min Act: -0.1700

Data Index: 1061165 (C4 (Web Text))

Max Activating Token Index: 968

Text #9

Max Range: 3.3879. Min Range: -3.3879

Max Act: 3.1206. Min Act: -0.1700

Data Index: 1320859 (C4 (Web Text))

Max Activating Token Index: 916

Text #10

Max Range: 3.3879. Min Range: -3.3879

Max Act: 3.1206. Min Act: -0.1700

Data Index: 1349492 (C4 (Web Text))

Max Activating Token Index: 635

Text #11

Max Range: 3.3879. Min Range: -3.3879

Max Act: 3.0891. Min Act: -0.1700

Data Index: 253458 (C4 (Web Text))

Max Activating Token Index: 385

Text #12

Max Range: 3.3879. Min Range: -3.3879

Max Act: 3.0891. Min Act: -0.1700

Data Index: 1014250 (C4 (Web Text))

Max Activating Token Index: 543

Text #13

Max Range: 3.3879. Min Range: -3.3879

Max Act: 3.0733. Min Act: -0.1700

Data Index: 951071 (C4 (Web Text))

Max Activating Token Index: 651

Text #14

Max Range: 3.3879. Min Range: -3.3879

Max Act: 3.0733. Min Act: -0.1700

Data Index: 1616261 (Python Code)

Max Activating Token Index: 76

Text #15

Max Range: 3.3879. Min Range: -3.3879

Max Act: 3.0891. Min Act: -0.1700

Data Index: 279085 (C4 (Web Text))

Max Activating Token Index: 341

Text #16

Max Range: 3.3879. Min Range: -3.3879

Max Act: 3.0575. Min Act: -0.1700

Data Index: 885737 (C4 (Web Text))

Max Activating Token Index: 748

Text #17

Max Range: 3.3879. Min Range: -3.3879

Max Act: 3.0733. Min Act: -0.1700

Data Index: 331751 (C4 (Web Text))

Max Activating Token Index: 854

Text #18

Max Range: 3.3879. Min Range: -3.3879

Max Act: 3.0733. Min Act: -0.1700

Data Index: 1180970 (C4 (Web Text))

Max Activating Token Index: 890

Text #19

Max Range: 3.3879. Min Range: -3.3879

Max Act: 3.1049. Min Act: -0.1700

Data Index: 806241 (C4 (Web Text))

Max Activating Token Index: 703