Model: GELU Model: 4 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1415 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('gelu-4l')

Text #0

Max Range: 3.9055. Min Range: -3.9055

Max Act: 3.9055. Min Act: -0.1700

Data Index: 1312872 (C4 (Web Text))

Max Activating Token Index: 772

Text #1

Max Range: 3.9055. Min Range: -3.9055

Max Act: 3.7178. Min Act: -0.1700

Data Index: 1315631 (C4 (Web Text))

Max Activating Token Index: 850

Text #2

Max Range: 3.9055. Min Range: -3.9055

Max Act: 3.7335. Min Act: -0.1700

Data Index: 635834 (C4 (Web Text))

Max Activating Token Index: 751

Text #3

Max Range: 3.9055. Min Range: -3.9055

Max Act: 3.7022. Min Act: -0.1700

Data Index: 821360 (C4 (Web Text))

Max Activating Token Index: 788

Text #4

Max Range: 3.9055. Min Range: -3.9055

Max Act: 3.7022. Min Act: -0.1700

Data Index: 856105 (C4 (Web Text))

Max Activating Token Index: 714

Text #5

Max Range: 3.9055. Min Range: -3.9055

Max Act: 3.6865. Min Act: -0.1700

Data Index: 188891 (C4 (Web Text))

Max Activating Token Index: 804

Text #6

Max Range: 3.9055. Min Range: -3.9055

Max Act: 3.6865. Min Act: -0.1700

Data Index: 1031088 (C4 (Web Text))

Max Activating Token Index: 477

Text #7

Max Range: 3.9055. Min Range: -3.9055

Max Act: 3.6709. Min Act: -0.1700

Data Index: 631706 (C4 (Web Text))

Max Activating Token Index: 899

Text #8

Max Range: 3.9055. Min Range: -3.9055

Max Act: 3.7178. Min Act: -0.1700

Data Index: 906227 (C4 (Web Text))

Max Activating Token Index: 461

Text #9

Max Range: 3.9055. Min Range: -3.9055

Max Act: 3.6709. Min Act: -0.1700

Data Index: 1105873 (C4 (Web Text))

Max Activating Token Index: 611

Text #10

Max Range: 3.9055. Min Range: -3.9055

Max Act: 3.6865. Min Act: -0.1700

Data Index: 216789 (C4 (Web Text))

Max Activating Token Index: 860

Text #11

Max Range: 3.9055. Min Range: -3.9055

Max Act: 3.6552. Min Act: -0.1700

Data Index: 1207944 (C4 (Web Text))

Max Activating Token Index: 475

Text #12

Max Range: 3.9055. Min Range: -3.9055

Max Act: 3.6709. Min Act: -0.1700

Data Index: 731531 (C4 (Web Text))

Max Activating Token Index: 702

Text #13

Max Range: 3.9055. Min Range: -3.9055

Max Act: 3.6396. Min Act: -0.1700

Data Index: 165028 (C4 (Web Text))

Max Activating Token Index: 789

Text #14

Max Range: 3.9055. Min Range: -3.9055

Max Act: 3.6552. Min Act: -0.1700

Data Index: 200976 (C4 (Web Text))

Max Activating Token Index: 708

Text #15

Max Range: 3.9055. Min Range: -3.9055

Max Act: 3.6865. Min Act: -0.1700

Data Index: 199397 (C4 (Web Text))

Max Activating Token Index: 971

Text #16

Max Range: 3.9055. Min Range: -3.9055

Max Act: 3.6709. Min Act: -0.1700

Data Index: 889363 (C4 (Web Text))

Max Activating Token Index: 788

Text #17

Max Range: 3.9055. Min Range: -3.9055

Max Act: 3.6552. Min Act: -0.1700

Data Index: 532328 (C4 (Web Text))

Max Activating Token Index: 859

Text #18

Max Range: 3.9055. Min Range: -3.9055

Max Act: 3.6552. Min Act: -0.1700

Data Index: 317893 (C4 (Web Text))

Max Activating Token Index: 259

Text #19

Max Range: 3.9055. Min Range: -3.9055

Max Act: 3.5926. Min Act: -0.1700

Data Index: 1292547 (C4 (Web Text))

Max Activating Token Index: 955