Model: GELU Model: 4 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 272 in Layer 3

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('gelu-4l')

Text #0

Max Range: 5.7397. Min Range: -5.7397

Max Act: 5.7397. Min Act: -0.1700

Data Index: 516923 (C4 (Web Text))

Max Activating Token Index: 750

Text #1

Max Range: 5.7397. Min Range: -5.7397

Max Act: 5.5834. Min Act: -0.1700

Data Index: 1276427 (C4 (Web Text))

Max Activating Token Index: 930

Text #2

Max Range: 5.7397. Min Range: -5.7397

Max Act: 5.5522. Min Act: -0.1700

Data Index: 59232 (C4 (Web Text))

Max Activating Token Index: 555

Text #3

Max Range: 5.7397. Min Range: -5.7397

Max Act: 5.6147. Min Act: -0.1700

Data Index: 307266 (C4 (Web Text))

Max Activating Token Index: 284

Text #4

Max Range: 5.7397. Min Range: -5.7397

Max Act: 5.5522. Min Act: -0.1700

Data Index: 1299146 (C4 (Web Text))

Max Activating Token Index: 844

Text #5

Max Range: 5.7397. Min Range: -5.7397

Max Act: 5.5209. Min Act: -0.1700

Data Index: 567934 (C4 (Web Text))

Max Activating Token Index: 653

Text #6

Max Range: 5.7397. Min Range: -5.7397

Max Act: 5.5209. Min Act: -0.1700

Data Index: 1176654 (C4 (Web Text))

Max Activating Token Index: 766

Text #7

Max Range: 5.7397. Min Range: -5.7397

Max Act: 5.4584. Min Act: -0.1700

Data Index: 270827 (C4 (Web Text))

Max Activating Token Index: 574

Text #8

Max Range: 5.7397. Min Range: -5.7397

Max Act: 5.4897. Min Act: -0.1700

Data Index: 662841 (C4 (Web Text))

Max Activating Token Index: 890

Text #9

Max Range: 5.7397. Min Range: -5.7397

Max Act: 5.4897. Min Act: -0.1700

Data Index: 1175118 (C4 (Web Text))

Max Activating Token Index: 745

Text #10

Max Range: 5.7397. Min Range: -5.7397

Max Act: 5.4272. Min Act: -0.1700

Data Index: 1530184 (Python Code)

Max Activating Token Index: 600

Text #11

Max Range: 5.7397. Min Range: -5.7397

Max Act: 5.3959. Min Act: -0.1700

Data Index: 674207 (C4 (Web Text))

Max Activating Token Index: 411

Text #12

Max Range: 5.7397. Min Range: -5.7397

Max Act: 5.4272. Min Act: -0.1700

Data Index: 573998 (C4 (Web Text))

Max Activating Token Index: 872

Text #13

Max Range: 5.7397. Min Range: -5.7397

Max Act: 5.3959. Min Act: -0.1700

Data Index: 447455 (C4 (Web Text))

Max Activating Token Index: 517

Text #14

Max Range: 5.7397. Min Range: -5.7397

Max Act: 5.3334. Min Act: -0.1700

Data Index: 107326 (C4 (Web Text))

Max Activating Token Index: 407

Text #15

Max Range: 5.7397. Min Range: -5.7397

Max Act: 5.3959. Min Act: -0.1700

Data Index: 87255 (C4 (Web Text))

Max Activating Token Index: 407

Text #16

Max Range: 5.7397. Min Range: -5.7397

Max Act: 5.3959. Min Act: -0.1700

Data Index: 64941 (C4 (Web Text))

Max Activating Token Index: 495

Text #17

Max Range: 5.7397. Min Range: -5.7397

Max Act: 5.3647. Min Act: -0.1700

Data Index: 285029 (C4 (Web Text))

Max Activating Token Index: 386

Text #18

Max Range: 5.7397. Min Range: -5.7397

Max Act: 5.3647. Min Act: -0.1700

Data Index: 1624598 (Python Code)

Max Activating Token Index: 526

Text #19

Max Range: 5.7397. Min Range: -5.7397

Max Act: 5.3647. Min Act: -0.1700

Data Index: 1607172 (Python Code)

Max Activating Token Index: 959