Model: GELU Model: 4 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 730 in Layer 0

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('gelu-4l')

Text #0

Max Range: 3.2619. Min Range: -3.2619

Max Act: 3.2619. Min Act: -0.1700

Data Index: 1188867 (C4 (Web Text))

Max Activating Token Index: 254

Text #1

Max Range: 3.2619. Min Range: -3.2619

Max Act: 3.2462. Min Act: -0.1700

Data Index: 370217 (C4 (Web Text))

Max Activating Token Index: 717

Text #2

Max Range: 3.2619. Min Range: -3.2619

Max Act: 3.1675. Min Act: -0.1700

Data Index: 1642511 (Python Code)

Max Activating Token Index: 5

Text #3

Max Range: 3.2619. Min Range: -3.2619

Max Act: 3.1675. Min Act: -0.1700

Data Index: 441614 (C4 (Web Text))

Max Activating Token Index: 557

Text #4

Max Range: 3.2619. Min Range: -3.2619

Max Act: 3.1360. Min Act: -0.1700

Data Index: 144792 (C4 (Web Text))

Max Activating Token Index: 922

Text #5

Max Range: 3.2619. Min Range: -3.2619

Max Act: 3.1202. Min Act: -0.1700

Data Index: 678951 (C4 (Web Text))

Max Activating Token Index: 4

Text #6

Max Range: 3.2619. Min Range: -3.2619

Max Act: 3.1202. Min Act: -0.1700

Data Index: 705255 (C4 (Web Text))

Max Activating Token Index: 354

Text #7

Max Range: 3.2619. Min Range: -3.2619

Max Act: 3.1202. Min Act: -0.1700

Data Index: 1117899 (C4 (Web Text))

Max Activating Token Index: 631

Text #8

Max Range: 3.2619. Min Range: -3.2619

Max Act: 3.1045. Min Act: -0.1700

Data Index: 814539 (C4 (Web Text))

Max Activating Token Index: 885

Text #9

Max Range: 3.2619. Min Range: -3.2619

Max Act: 3.0887. Min Act: -0.1700

Data Index: 1132020 (C4 (Web Text))

Max Activating Token Index: 3

Text #10

Max Range: 3.2619. Min Range: -3.2619

Max Act: 3.0887. Min Act: -0.1700

Data Index: 1097983 (C4 (Web Text))

Max Activating Token Index: 3

Text #11

Max Range: 3.2619. Min Range: -3.2619

Max Act: 3.0414. Min Act: -0.1700

Data Index: 1149397 (C4 (Web Text))

Max Activating Token Index: 799

Text #12

Max Range: 3.2619. Min Range: -3.2619

Max Act: 3.0887. Min Act: -0.1700

Data Index: 1127594 (C4 (Web Text))

Max Activating Token Index: 490

Text #13

Max Range: 3.2619. Min Range: -3.2619

Max Act: 3.0414. Min Act: -0.1700

Data Index: 933170 (C4 (Web Text))

Max Activating Token Index: 962

Text #14

Max Range: 3.2619. Min Range: -3.2619

Max Act: 3.0729. Min Act: -0.1700

Data Index: 1342077 (C4 (Web Text))

Max Activating Token Index: 582

Text #15

Max Range: 3.2619. Min Range: -3.2619

Max Act: 3.0414. Min Act: -0.1700

Data Index: 881565 (C4 (Web Text))

Max Activating Token Index: 439

Text #16

Max Range: 3.2619. Min Range: -3.2619

Max Act: 3.0572. Min Act: -0.1700

Data Index: 222250 (C4 (Web Text))

Max Activating Token Index: 258

Text #17

Max Range: 3.2619. Min Range: -3.2619

Max Act: 3.0572. Min Act: -0.1700

Data Index: 120403 (C4 (Web Text))

Max Activating Token Index: 406

Text #18

Max Range: 3.2619. Min Range: -3.2619

Max Act: 3.0572. Min Act: -0.1700

Data Index: 469681 (C4 (Web Text))

Max Activating Token Index: 400

Text #19

Max Range: 3.2619. Min Range: -3.2619

Max Act: 3.0572. Min Act: -0.1700

Data Index: 91932 (C4 (Web Text))

Max Activating Token Index: 343