Model: GELU Model: 3 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1544 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('gelu-3l')

Text #0

Max Range: 5.9222. Min Range: -5.9222

Max Act: 5.9222. Min Act: -0.1700

Data Index: 161713 (C4 (Web Text))

Max Activating Token Index: 983

Text #1

Max Range: 5.9222. Min Range: -5.9222

Max Act: 5.8910. Min Act: -0.1700

Data Index: 649340 (C4 (Web Text))

Max Activating Token Index: 203

Text #2

Max Range: 5.9222. Min Range: -5.9222

Max Act: 5.7972. Min Act: -0.1699

Data Index: 1101693 (C4 (Web Text))

Max Activating Token Index: 266

Text #3

Max Range: 5.9222. Min Range: -5.9222

Max Act: 5.7972. Min Act: -0.1700

Data Index: 278238 (C4 (Web Text))

Max Activating Token Index: 87

Text #4

Max Range: 5.9222. Min Range: -5.9222

Max Act: 5.7347. Min Act: -0.1700

Data Index: 1214132 (C4 (Web Text))

Max Activating Token Index: 504

Text #5

Max Range: 5.9222. Min Range: -5.9222

Max Act: 5.7347. Min Act: -0.1700

Data Index: 710835 (C4 (Web Text))

Max Activating Token Index: 1003

Text #6

Max Range: 5.9222. Min Range: -5.9222

Max Act: 5.7347. Min Act: -0.1700

Data Index: 1224905 (C4 (Web Text))

Max Activating Token Index: 848

Text #7

Max Range: 5.9222. Min Range: -5.9222

Max Act: 5.7035. Min Act: -0.1700

Data Index: 127795 (C4 (Web Text))

Max Activating Token Index: 58

Text #8

Max Range: 5.9222. Min Range: -5.9222

Max Act: 5.7660. Min Act: -0.1700

Data Index: 1125539 (C4 (Web Text))

Max Activating Token Index: 736

Text #9

Max Range: 5.9222. Min Range: -5.9222

Max Act: 5.7347. Min Act: -0.1700

Data Index: 365588 (C4 (Web Text))

Max Activating Token Index: 364

Text #10

Max Range: 5.9222. Min Range: -5.9222

Max Act: 5.7660. Min Act: -0.1700

Data Index: 1284554 (C4 (Web Text))

Max Activating Token Index: 789

Text #11

Max Range: 5.9222. Min Range: -5.9222

Max Act: 5.7035. Min Act: -0.1700

Data Index: 1250469 (C4 (Web Text))

Max Activating Token Index: 35

Text #12

Max Range: 5.9222. Min Range: -5.9222

Max Act: 5.7972. Min Act: -0.1700

Data Index: 107185 (C4 (Web Text))

Max Activating Token Index: 333

Text #13

Max Range: 5.9222. Min Range: -5.9222

Max Act: 5.7347. Min Act: -0.1700

Data Index: 743967 (C4 (Web Text))

Max Activating Token Index: 710

Text #14

Max Range: 5.9222. Min Range: -5.9222

Max Act: 5.6410. Min Act: -0.1700

Data Index: 1018708 (C4 (Web Text))

Max Activating Token Index: 888

Text #15

Max Range: 5.9222. Min Range: -5.9222

Max Act: 5.6722. Min Act: -0.1700

Data Index: 585512 (C4 (Web Text))

Max Activating Token Index: 648

Text #16

Max Range: 5.9222. Min Range: -5.9222

Max Act: 5.6410. Min Act: -0.1700

Data Index: 49089 (C4 (Web Text))

Max Activating Token Index: 416

Text #17

Max Range: 5.9222. Min Range: -5.9222

Max Act: 5.6722. Min Act: -0.1700

Data Index: 1201372 (C4 (Web Text))

Max Activating Token Index: 93

Text #18

Max Range: 5.9222. Min Range: -5.9222

Max Act: 5.6722. Min Act: -0.1699

Data Index: 596102 (C4 (Web Text))

Max Activating Token Index: 335

Text #19

Max Range: 5.9222. Min Range: -5.9222

Max Act: 5.5160. Min Act: -0.1700

Data Index: 49734 (C4 (Web Text))

Max Activating Token Index: 934