Model: GELU Model: 4 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 439 in Layer 3

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('gelu-4l')

Text #0

Max Range: 8.5986. Min Range: -8.5986

Max Act: 8.5986. Min Act: -0.1700

Data Index: 1131870 (C4 (Web Text))

Max Activating Token Index: 503

Text #1

Max Range: 8.5986. Min Range: -8.5986

Max Act: 8.3486. Min Act: -0.1700

Data Index: 212649 (C4 (Web Text))

Max Activating Token Index: 435

Text #2

Max Range: 8.5986. Min Range: -8.5986

Max Act: 8.3486. Min Act: -0.1700

Data Index: 617174 (C4 (Web Text))

Max Activating Token Index: 993

Text #3

Max Range: 8.5986. Min Range: -8.5986

Max Act: 8.2861. Min Act: -0.1700

Data Index: 1025735 (C4 (Web Text))

Max Activating Token Index: 729

Text #4

Max Range: 8.5986. Min Range: -8.5986

Max Act: 8.2861. Min Act: -0.1700

Data Index: 491780 (C4 (Web Text))

Max Activating Token Index: 171

Text #5

Max Range: 8.5986. Min Range: -8.5986

Max Act: 8.2236. Min Act: -0.1700

Data Index: 604492 (C4 (Web Text))

Max Activating Token Index: 415

Text #6

Max Range: 8.5986. Min Range: -8.5986

Max Act: 8.2236. Min Act: -0.1700

Data Index: 1271736 (C4 (Web Text))

Max Activating Token Index: 446

Text #7

Max Range: 8.5986. Min Range: -8.5986

Max Act: 8.2236. Min Act: -0.1700

Data Index: 1187852 (C4 (Web Text))

Max Activating Token Index: 228

Text #8

Max Range: 8.5986. Min Range: -8.5986

Max Act: 8.1611. Min Act: -0.1700

Data Index: 1285527 (C4 (Web Text))

Max Activating Token Index: 681

Text #9

Max Range: 8.5986. Min Range: -8.5986

Max Act: 8.1611. Min Act: -0.1700

Data Index: 1292388 (C4 (Web Text))

Max Activating Token Index: 646

Text #10

Max Range: 8.5986. Min Range: -8.5986

Max Act: 8.0986. Min Act: -0.1700

Data Index: 1100216 (C4 (Web Text))

Max Activating Token Index: 134

Text #11

Max Range: 8.5986. Min Range: -8.5986

Max Act: 8.0986. Min Act: -0.1700

Data Index: 487978 (C4 (Web Text))

Max Activating Token Index: 782

Text #12

Max Range: 8.5986. Min Range: -8.5986

Max Act: 8.2236. Min Act: -0.1700

Data Index: 888470 (C4 (Web Text))

Max Activating Token Index: 429

Text #13

Max Range: 8.5986. Min Range: -8.5986

Max Act: 8.0361. Min Act: -0.1699

Data Index: 576362 (C4 (Web Text))

Max Activating Token Index: 398

Text #14

Max Range: 8.5986. Min Range: -8.5986

Max Act: 8.0986. Min Act: -0.1700

Data Index: 1155347 (C4 (Web Text))

Max Activating Token Index: 129

Text #15

Max Range: 8.5986. Min Range: -8.5986

Max Act: 8.0986. Min Act: -0.1700

Data Index: 1114229 (C4 (Web Text))

Max Activating Token Index: 828

Text #16

Max Range: 8.5986. Min Range: -8.5986

Max Act: 7.9736. Min Act: -0.1700

Data Index: 41382 (C4 (Web Text))

Max Activating Token Index: 480

Text #17

Max Range: 8.5986. Min Range: -8.5986

Max Act: 7.9736. Min Act: -0.1700

Data Index: 460525 (C4 (Web Text))

Max Activating Token Index: 548

Text #18

Max Range: 8.5986. Min Range: -8.5986

Max Act: 8.0986. Min Act: -0.1700

Data Index: 752314 (C4 (Web Text))

Max Activating Token Index: 435

Text #19

Max Range: 8.5986. Min Range: -8.5986

Max Act: 8.0361. Min Act: -0.1699

Data Index: 1110756 (C4 (Web Text))

Max Activating Token Index: 142