Model: GELU Model: 4 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 204 in Layer 3

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('gelu-4l')

Text #0

Max Range: 5.3968. Min Range: -5.3968

Max Act: 5.3968. Min Act: -0.1700

Data Index: 1001389 (C4 (Web Text))

Max Activating Token Index: 1001

Text #1

Max Range: 5.3968. Min Range: -5.3968

Max Act: 5.3343. Min Act: -0.1700

Data Index: 326182 (C4 (Web Text))

Max Activating Token Index: 1011

Text #2

Max Range: 5.3968. Min Range: -5.3968

Max Act: 5.2718. Min Act: -0.1700

Data Index: 1006448 (C4 (Web Text))

Max Activating Token Index: 6

Text #3

Max Range: 5.3968. Min Range: -5.3968

Max Act: 5.1781. Min Act: -0.1700

Data Index: 324959 (C4 (Web Text))

Max Activating Token Index: 988

Text #4

Max Range: 5.3968. Min Range: -5.3968

Max Act: 5.2093. Min Act: -0.1700

Data Index: 576737 (C4 (Web Text))

Max Activating Token Index: 1023

Text #5

Max Range: 5.3968. Min Range: -5.3968

Max Act: 5.2406. Min Act: -0.1700

Data Index: 1278234 (C4 (Web Text))

Max Activating Token Index: 556

Text #6

Max Range: 5.3968. Min Range: -5.3968

Max Act: 5.1781. Min Act: -0.1700

Data Index: 1155696 (C4 (Web Text))

Max Activating Token Index: 898

Text #7

Max Range: 5.3968. Min Range: -5.3968

Max Act: 5.1781. Min Act: -0.1700

Data Index: 923292 (C4 (Web Text))

Max Activating Token Index: 178

Text #8

Max Range: 5.3968. Min Range: -5.3968

Max Act: 5.1781. Min Act: -0.1700

Data Index: 1256917 (C4 (Web Text))

Max Activating Token Index: 160

Text #9

Max Range: 5.3968. Min Range: -5.3968

Max Act: 5.1781. Min Act: -0.1700

Data Index: 1244501 (C4 (Web Text))

Max Activating Token Index: 592

Text #10

Max Range: 5.3968. Min Range: -5.3968

Max Act: 5.1781. Min Act: -0.1700

Data Index: 56609 (C4 (Web Text))

Max Activating Token Index: 320

Text #11

Max Range: 5.3968. Min Range: -5.3968

Max Act: 5.0843. Min Act: -0.1700

Data Index: 1195479 (C4 (Web Text))

Max Activating Token Index: 447

Text #12

Max Range: 5.3968. Min Range: -5.3968

Max Act: 5.1468. Min Act: -0.1700

Data Index: 849929 (C4 (Web Text))

Max Activating Token Index: 791

Text #13

Max Range: 5.3968. Min Range: -5.3968

Max Act: 5.0531. Min Act: -0.1700

Data Index: 471403 (C4 (Web Text))

Max Activating Token Index: 686

Text #14

Max Range: 5.3968. Min Range: -5.3968

Max Act: 5.0843. Min Act: -0.1700

Data Index: 885242 (C4 (Web Text))

Max Activating Token Index: 487

Text #15

Max Range: 5.3968. Min Range: -5.3968

Max Act: 5.1156. Min Act: -0.1700

Data Index: 748787 (C4 (Web Text))

Max Activating Token Index: 796

Text #16

Max Range: 5.3968. Min Range: -5.3968

Max Act: 5.1156. Min Act: -0.1700

Data Index: 1112425 (C4 (Web Text))

Max Activating Token Index: 181

Text #17

Max Range: 5.3968. Min Range: -5.3968

Max Act: 5.1156. Min Act: -0.1700

Data Index: 453450 (C4 (Web Text))

Max Activating Token Index: 882

Text #18

Max Range: 5.3968. Min Range: -5.3968

Max Act: 5.1468. Min Act: -0.1700

Data Index: 836468 (C4 (Web Text))

Max Activating Token Index: 669

Text #19

Max Range: 5.3968. Min Range: -5.3968

Max Act: 5.1156. Min Act: -0.1700

Data Index: 1222719 (C4 (Web Text))

Max Activating Token Index: 229