Model: GELU Model: 1 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1305 in Layer 0

Load this data into an Interactive Neuroscope

See Documentation here

Hooked Transformer Loading: HookedTransformer.from_pretrained('gelu-1l')

Click the toggle to see the full text!

Text #0

Max Range: 2.9173. Min Range: -2.9173

Max Act: 2.9173. Min Act: -0.1343

Data Index: 577786 (C4 (Web Text))

Max Activating Token Index: 460

Text #1

Max Range: 2.9173. Min Range: -2.9173

Max Act: 2.8295. Min Act: -0.1700

Data Index: 628137 (C4 (Web Text))

Max Activating Token Index: 1008

Text #2

Max Range: 2.9173. Min Range: -2.9173

Max Act: 2.7868. Min Act: -0.1668

Data Index: 1386229 (Python Code)

Max Activating Token Index: 308

Text #3

Max Range: 2.9173. Min Range: -2.9173

Max Act: 2.7224. Min Act: -0.1147

Data Index: 1051048 (C4 (Web Text))

Max Activating Token Index: 282

Text #4

Max Range: 2.9173. Min Range: -2.9173

Max Act: 2.7043. Min Act: -0.1700

Data Index: 289162 (C4 (Web Text))

Max Activating Token Index: 802

Text #5

Max Range: 2.9173. Min Range: -2.9173

Max Act: 2.6835. Min Act: -0.1579

Data Index: 1592810 (Python Code)

Max Activating Token Index: 981

Text #6

Max Range: 2.9173. Min Range: -2.9173

Max Act: 2.6898. Min Act: -0.1694

Data Index: 1497012 (Python Code)

Max Activating Token Index: 409

Text #7

Max Range: 2.9173. Min Range: -2.9173

Max Act: 2.6648. Min Act: -0.1699

Data Index: 1145309 (C4 (Web Text))

Max Activating Token Index: 394

Text #8

Max Range: 2.9173. Min Range: -2.9173

Max Act: 2.6555. Min Act: -0.1638

Data Index: 1397799 (Python Code)

Max Activating Token Index: 340

Text #9

Max Range: 2.9173. Min Range: -2.9173

Max Act: 2.6624. Min Act: -0.1700

Data Index: 1406081 (Python Code)

Max Activating Token Index: 484

Text #10

Max Range: 2.9173. Min Range: -2.9173

Max Act: 2.6471. Min Act: -0.1700

Data Index: 294915 (C4 (Web Text))

Max Activating Token Index: 410

Text #11

Max Range: 2.9173. Min Range: -2.9173

Max Act: 2.6330. Min Act: -0.1570

Data Index: 1497281 (Python Code)

Max Activating Token Index: 679

Text #12

Max Range: 2.9173. Min Range: -2.9173

Max Act: 2.6377. Min Act: -0.1700

Data Index: 521205 (C4 (Web Text))

Max Activating Token Index: 249

Text #13

Max Range: 2.9173. Min Range: -2.9173

Max Act: 2.6158. Min Act: -0.1700

Data Index: 1510372 (Python Code)

Max Activating Token Index: 1023

Text #14

Max Range: 2.9173. Min Range: -2.9173

Max Act: 2.6264. Min Act: -0.1700

Data Index: 1475716 (Python Code)

Max Activating Token Index: 883

Text #15

Max Range: 2.9173. Min Range: -2.9173

Max Act: 2.6312. Min Act: -0.1395

Data Index: 1635567 (Python Code)

Max Activating Token Index: 897

Text #16

Max Range: 2.9173. Min Range: -2.9173

Max Act: 2.6108. Min Act: -0.1700

Data Index: 55561 (C4 (Web Text))

Max Activating Token Index: 161

Text #17

Max Range: 2.9173. Min Range: -2.9173

Max Act: 2.6342. Min Act: -0.1683

Data Index: 800655 (C4 (Web Text))

Max Activating Token Index: 928

Text #18

Max Range: 2.9173. Min Range: -2.9173

Max Act: 2.6066. Min Act: -0.1700

Data Index: 219196 (C4 (Web Text))

Max Activating Token Index: 355

Text #19

Max Range: 2.9173. Min Range: -2.9173

Max Act: 2.5982. Min Act: -0.1700

Data Index: 819543 (C4 (Web Text))

Max Activating Token Index: 271