Model: GELU Model: 3 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1589 in Layer 0

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('gelu-3l')

Text #0

Max Range: 5.7732. Min Range: -5.7732

Max Act: 5.7732. Min Act: -0.1699

Data Index: 1173895 (C4 (Web Text))

Max Activating Token Index: 502

Text #1

Max Range: 5.7732. Min Range: -5.7732

Max Act: 5.7420. Min Act: -0.1700

Data Index: 1339434 (C4 (Web Text))

Max Activating Token Index: 165

Text #2

Max Range: 5.7732. Min Range: -5.7732

Max Act: 5.7107. Min Act: -0.1700

Data Index: 115342 (C4 (Web Text))

Max Activating Token Index: 1016

Text #3

Max Range: 5.7732. Min Range: -5.7732

Max Act: 5.6795. Min Act: -0.1700

Data Index: 872006 (C4 (Web Text))

Max Activating Token Index: 113

Text #4

Max Range: 5.7732. Min Range: -5.7732

Max Act: 5.6795. Min Act: -0.1700

Data Index: 108031 (C4 (Web Text))

Max Activating Token Index: 113

Text #5

Max Range: 5.7732. Min Range: -5.7732

Max Act: 5.6170. Min Act: -0.1700

Data Index: 56617 (C4 (Web Text))

Max Activating Token Index: 207

Text #6

Max Range: 5.7732. Min Range: -5.7732

Max Act: 5.5857. Min Act: -0.1695

Data Index: 138536 (C4 (Web Text))

Max Activating Token Index: 394

Text #7

Max Range: 5.7732. Min Range: -5.7732

Max Act: 5.5232. Min Act: -0.1635

Data Index: 388368 (C4 (Web Text))

Max Activating Token Index: 1023

Text #8

Max Range: 5.7732. Min Range: -5.7732

Max Act: 5.4607. Min Act: -0.1700

Data Index: 1065199 (C4 (Web Text))

Max Activating Token Index: 108

Text #9

Max Range: 5.7732. Min Range: -5.7732

Max Act: 5.3357. Min Act: -0.1697

Data Index: 524660 (C4 (Web Text))

Max Activating Token Index: 736

Text #10

Max Range: 5.7732. Min Range: -5.7732

Max Act: 5.3045. Min Act: -0.1700

Data Index: 401134 (C4 (Web Text))

Max Activating Token Index: 408

Text #11

Max Range: 5.7732. Min Range: -5.7732

Max Act: 5.3045. Min Act: -0.1615

Data Index: 766730 (C4 (Web Text))

Max Activating Token Index: 954

Text #12

Max Range: 5.7732. Min Range: -5.7732

Max Act: 5.3045. Min Act: -0.1678

Data Index: 244706 (C4 (Web Text))

Max Activating Token Index: 935

Text #13

Max Range: 5.7732. Min Range: -5.7732

Max Act: 5.3045. Min Act: -0.1313

Data Index: 373896 (C4 (Web Text))

Max Activating Token Index: 429

Text #14

Max Range: 5.7732. Min Range: -5.7732

Max Act: 5.2732. Min Act: -0.1700

Data Index: 270082 (C4 (Web Text))

Max Activating Token Index: 970

Text #15

Max Range: 5.7732. Min Range: -5.7732

Max Act: 5.2732. Min Act: -0.1700

Data Index: 951723 (C4 (Web Text))

Max Activating Token Index: 36

Text #16

Max Range: 5.7732. Min Range: -5.7732

Max Act: 5.2732. Min Act: -0.1700

Data Index: 328582 (C4 (Web Text))

Max Activating Token Index: 246

Text #17

Max Range: 5.7732. Min Range: -5.7732

Max Act: 5.2420. Min Act: -0.1584

Data Index: 1251810 (C4 (Web Text))

Max Activating Token Index: 1004

Text #18

Max Range: 5.7732. Min Range: -5.7732

Max Act: 5.2107. Min Act: -0.1700

Data Index: 164811 (C4 (Web Text))

Max Activating Token Index: 980

Text #19

Max Range: 5.7732. Min Range: -5.7732

Max Act: 5.2420. Min Act: -0.1700

Data Index: 1048469 (C4 (Web Text))

Max Activating Token Index: 515