Model: GELU Model: 4 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1759 in Layer 0

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('gelu-4l')

Text #0

Max Range: 1.9861. Min Range: -1.9861

Max Act: 1.9861. Min Act: -0.0677

Data Index: 76368 (C4 (Web Text))

Max Activating Token Index: 613

Text #1

Max Range: 1.9861. Min Range: -1.9861

Max Act: 1.9776. Min Act: -0.0000

Data Index: 1516391 (Python Code)

Max Activating Token Index: 613

Text #2

Max Range: 1.9861. Min Range: -1.9861

Max Act: 1.9776. Min Act: 0.0050

Data Index: 864259 (C4 (Web Text))

Max Activating Token Index: 613

Text #3

Max Range: 1.9861. Min Range: -1.9861

Max Act: 1.9861. Min Act: -0.1190

Data Index: 1429814 (Python Code)

Max Activating Token Index: 613

Text #4

Max Range: 1.9861. Min Range: -1.9861

Max Act: 1.9607. Min Act: -0.0725

Data Index: 1631062 (Python Code)

Max Activating Token Index: 613

Text #5

Max Range: 1.9861. Min Range: -1.9861

Max Act: 1.9607. Min Act: -0.0743

Data Index: 1394686 (Python Code)

Max Activating Token Index: 613

Text #6

Max Range: 1.9861. Min Range: -1.9861

Max Act: 1.9692. Min Act: -0.0739

Data Index: 1093118 (C4 (Web Text))

Max Activating Token Index: 818

Text #7

Max Range: 1.9861. Min Range: -1.9861

Max Act: 1.9607. Min Act: -0.1214

Data Index: 1347033 (C4 (Web Text))

Max Activating Token Index: 818

Text #8

Max Range: 1.9861. Min Range: -1.9861

Max Act: 1.9522. Min Act: -0.0113

Data Index: 1498362 (Python Code)

Max Activating Token Index: 613

Text #9

Max Range: 1.9861. Min Range: -1.9861

Max Act: 1.9522. Min Act: -0.0746

Data Index: 895217 (C4 (Web Text))

Max Activating Token Index: 613

Text #10

Max Range: 1.9861. Min Range: -1.9861

Max Act: 1.9437. Min Act: -0.1073

Data Index: 972175 (C4 (Web Text))

Max Activating Token Index: 818

Text #11

Max Range: 1.9861. Min Range: -1.9861

Max Act: 1.9437. Min Act: -0.0169

Data Index: 1472197 (Python Code)

Max Activating Token Index: 613

Text #12

Max Range: 1.9861. Min Range: -1.9861

Max Act: 1.9522. Min Act: 0.0317

Data Index: 1472802 (Python Code)

Max Activating Token Index: 613

Text #13

Max Range: 1.9861. Min Range: -1.9861

Max Act: 1.9437. Min Act: -0.0397

Data Index: 1611126 (Python Code)

Max Activating Token Index: 613

Text #14

Max Range: 1.9861. Min Range: -1.9861

Max Act: 1.9437. Min Act: -0.0536

Data Index: 1385486 (Python Code)

Max Activating Token Index: 613

Text #15

Max Range: 1.9861. Min Range: -1.9861

Max Act: 1.9437. Min Act: -0.0103

Data Index: 1393896 (Python Code)

Max Activating Token Index: 499

Text #16

Max Range: 1.9861. Min Range: -1.9861

Max Act: 1.9522. Min Act: -0.0062

Data Index: 1589443 (Python Code)

Max Activating Token Index: 613

Text #17

Max Range: 1.9861. Min Range: -1.9861

Max Act: 1.9437. Min Act: -0.1252

Data Index: 1382452 (Python Code)

Max Activating Token Index: 613

Text #18

Max Range: 1.9861. Min Range: -1.9861

Max Act: 1.9353. Min Act: -0.0079

Data Index: 1541615 (Python Code)

Max Activating Token Index: 613

Text #19

Max Range: 1.9861. Min Range: -1.9861

Max Act: 1.9437. Min Act: -0.0330

Data Index: 1650904 (Python Code)

Max Activating Token Index: 613