Model: GELU Model: 3 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1674 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('gelu-3l')

Text #0

Max Range: 5.4359. Min Range: -5.4359

Max Act: 5.4359. Min Act: -0.1700

Data Index: 703925 (C4 (Web Text))

Max Activating Token Index: 282

Text #1

Max Range: 5.4359. Min Range: -5.4359

Max Act: 5.3109. Min Act: -0.1700

Data Index: 121678 (C4 (Web Text))

Max Activating Token Index: 299

Text #2

Max Range: 5.4359. Min Range: -5.4359

Max Act: 5.3109. Min Act: -0.1700

Data Index: 1273321 (C4 (Web Text))

Max Activating Token Index: 209

Text #3

Max Range: 5.4359. Min Range: -5.4359

Max Act: 5.2171. Min Act: -0.1700

Data Index: 33930 (C4 (Web Text))

Max Activating Token Index: 302

Text #4

Max Range: 5.4359. Min Range: -5.4359

Max Act: 5.1859. Min Act: -0.1700

Data Index: 209918 (C4 (Web Text))

Max Activating Token Index: 880

Text #5

Max Range: 5.4359. Min Range: -5.4359

Max Act: 5.2171. Min Act: -0.1700

Data Index: 240854 (C4 (Web Text))

Max Activating Token Index: 100

Text #6

Max Range: 5.4359. Min Range: -5.4359

Max Act: 5.1546. Min Act: -0.1700

Data Index: 492512 (C4 (Web Text))

Max Activating Token Index: 544

Text #7

Max Range: 5.4359. Min Range: -5.4359

Max Act: 5.1546. Min Act: -0.1700

Data Index: 1053969 (C4 (Web Text))

Max Activating Token Index: 283

Text #8

Max Range: 5.4359. Min Range: -5.4359

Max Act: 5.1546. Min Act: -0.1700

Data Index: 240144 (C4 (Web Text))

Max Activating Token Index: 35

Text #9

Max Range: 5.4359. Min Range: -5.4359

Max Act: 5.1859. Min Act: -0.1700

Data Index: 577406 (C4 (Web Text))

Max Activating Token Index: 939

Text #10

Max Range: 5.4359. Min Range: -5.4359

Max Act: 5.1859. Min Act: -0.1700

Data Index: 339345 (C4 (Web Text))

Max Activating Token Index: 668

Text #11

Max Range: 5.4359. Min Range: -5.4359

Max Act: 5.1234. Min Act: -0.1700

Data Index: 573940 (C4 (Web Text))

Max Activating Token Index: 245

Text #12

Max Range: 5.4359. Min Range: -5.4359

Max Act: 5.0921. Min Act: -0.1700

Data Index: 1242872 (C4 (Web Text))

Max Activating Token Index: 115

Text #13

Max Range: 5.4359. Min Range: -5.4359

Max Act: 5.0609. Min Act: -0.1700

Data Index: 331390 (C4 (Web Text))

Max Activating Token Index: 807

Text #14

Max Range: 5.4359. Min Range: -5.4359

Max Act: 5.1234. Min Act: -0.1700

Data Index: 618988 (C4 (Web Text))

Max Activating Token Index: 51

Text #15

Max Range: 5.4359. Min Range: -5.4359

Max Act: 5.0921. Min Act: -0.1700

Data Index: 679530 (C4 (Web Text))

Max Activating Token Index: 263

Text #16

Max Range: 5.4359. Min Range: -5.4359

Max Act: 5.1234. Min Act: -0.1700

Data Index: 1277989 (C4 (Web Text))

Max Activating Token Index: 14

Text #17

Max Range: 5.4359. Min Range: -5.4359

Max Act: 5.1234. Min Act: -0.1700

Data Index: 377716 (C4 (Web Text))

Max Activating Token Index: 57

Text #18

Max Range: 5.4359. Min Range: -5.4359

Max Act: 5.0609. Min Act: -0.1700

Data Index: 376036 (C4 (Web Text))

Max Activating Token Index: 284

Text #19

Max Range: 5.4359. Min Range: -5.4359

Max Act: 5.0609. Min Act: -0.1700

Data Index: 183807 (C4 (Web Text))

Max Activating Token Index: 231