Model: GELU Model: 4 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 373 in Layer 3

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('gelu-4l')

Text #0

Max Range: 5.1077. Min Range: -5.1077

Max Act: 5.1077. Min Act: -0.1700

Data Index: 1387771 (Python Code)

Max Activating Token Index: 928

Text #1

Max Range: 5.1077. Min Range: -5.1077

Max Act: 4.7640. Min Act: -0.1700

Data Index: 1549308 (Python Code)

Max Activating Token Index: 302

Text #2

Max Range: 5.1077. Min Range: -5.1077

Max Act: 4.7327. Min Act: -0.1700

Data Index: 1505282 (Python Code)

Max Activating Token Index: 395

Text #3

Max Range: 5.1077. Min Range: -5.1077

Max Act: 4.5921. Min Act: -0.1700

Data Index: 437764 (C4 (Web Text))

Max Activating Token Index: 950

Text #4

Max Range: 5.1077. Min Range: -5.1077

Max Act: 4.6234. Min Act: -0.1700

Data Index: 1511179 (Python Code)

Max Activating Token Index: 619

Text #5

Max Range: 5.1077. Min Range: -5.1077

Max Act: 4.5609. Min Act: -0.1700

Data Index: 762563 (C4 (Web Text))

Max Activating Token Index: 610

Text #6

Max Range: 5.1077. Min Range: -5.1077

Max Act: 4.6234. Min Act: -0.1700

Data Index: 1641927 (Python Code)

Max Activating Token Index: 668

Text #7

Max Range: 5.1077. Min Range: -5.1077

Max Act: 4.6077. Min Act: -0.1700

Data Index: 913576 (C4 (Web Text))

Max Activating Token Index: 713

Text #8

Max Range: 5.1077. Min Range: -5.1077

Max Act: 4.5452. Min Act: -0.1700

Data Index: 1605548 (Python Code)

Max Activating Token Index: 308

Text #9

Max Range: 5.1077. Min Range: -5.1077

Max Act: 4.5296. Min Act: -0.1700

Data Index: 1637509 (Python Code)

Max Activating Token Index: 200

Text #10

Max Range: 5.1077. Min Range: -5.1077

Max Act: 4.4984. Min Act: -0.1700

Data Index: 1498562 (Python Code)

Max Activating Token Index: 556

Text #11

Max Range: 5.1077. Min Range: -5.1077

Max Act: 4.5296. Min Act: -0.1700

Data Index: 1388006 (Python Code)

Max Activating Token Index: 875

Text #12

Max Range: 5.1077. Min Range: -5.1077

Max Act: 4.5140. Min Act: -0.1700

Data Index: 1241498 (C4 (Web Text))

Max Activating Token Index: 166

Text #13

Max Range: 5.1077. Min Range: -5.1077

Max Act: 4.4515. Min Act: -0.1700

Data Index: 1455251 (Python Code)

Max Activating Token Index: 183

Text #14

Max Range: 5.1077. Min Range: -5.1077

Max Act: 4.4827. Min Act: -0.1700

Data Index: 304444 (C4 (Web Text))

Max Activating Token Index: 578

Text #15

Max Range: 5.1077. Min Range: -5.1077

Max Act: 4.4202. Min Act: -0.1700

Data Index: 779584 (C4 (Web Text))

Max Activating Token Index: 314

Text #16

Max Range: 5.1077. Min Range: -5.1077

Max Act: 4.4515. Min Act: -0.1700

Data Index: 1562727 (Python Code)

Max Activating Token Index: 308

Text #17

Max Range: 5.1077. Min Range: -5.1077

Max Act: 4.4046. Min Act: -0.1700

Data Index: 1276449 (C4 (Web Text))

Max Activating Token Index: 99

Text #18

Max Range: 5.1077. Min Range: -5.1077

Max Act: 4.4202. Min Act: -0.1700

Data Index: 1644610 (Python Code)

Max Activating Token Index: 991

Text #19

Max Range: 5.1077. Min Range: -5.1077

Max Act: 4.3890. Min Act: -0.1700

Data Index: 780250 (C4 (Web Text))

Max Activating Token Index: 915