Model: GELU Model: 4 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 116 in Layer 3

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('gelu-4l')

Text #0

Max Range: 9.5802. Min Range: -9.5802

Max Act: 9.4865. Min Act: -0.1699

Data Index: 498643 (C4 (Web Text))

Max Activating Token Index: 909

Text #1

Max Range: 9.5802. Min Range: -9.5802

Max Act: 9.5802. Min Act: -0.1699

Data Index: 174135 (C4 (Web Text))

Max Activating Token Index: 898

Text #2

Max Range: 9.5802. Min Range: -9.5802

Max Act: 9.5177. Min Act: -0.1699

Data Index: 320098 (C4 (Web Text))

Max Activating Token Index: 611

Text #3

Max Range: 9.5802. Min Range: -9.5802

Max Act: 9.2990. Min Act: -0.1699

Data Index: 586823 (C4 (Web Text))

Max Activating Token Index: 785

Text #4

Max Range: 9.5802. Min Range: -9.5802

Max Act: 9.1427. Min Act: -0.1699

Data Index: 1356478 (C4 (Web Text))

Max Activating Token Index: 804

Text #5

Max Range: 9.5802. Min Range: -9.5802

Max Act: 9.3302. Min Act: -0.1699

Data Index: 536039 (C4 (Web Text))

Max Activating Token Index: 1006

Text #6

Max Range: 9.5802. Min Range: -9.5802

Max Act: 9.0802. Min Act: -0.1699

Data Index: 189718 (C4 (Web Text))

Max Activating Token Index: 564

Text #7

Max Range: 9.5802. Min Range: -9.5802

Max Act: 9.0802. Min Act: -0.1699

Data Index: 1475555 (Python Code)

Max Activating Token Index: 254

Text #8

Max Range: 9.5802. Min Range: -9.5802

Max Act: 8.8302. Min Act: -0.1699

Data Index: 1418175 (Python Code)

Max Activating Token Index: 447

Text #9

Max Range: 9.5802. Min Range: -9.5802

Max Act: 8.7677. Min Act: -0.1699

Data Index: 510048 (C4 (Web Text))

Max Activating Token Index: 9

Text #10

Max Range: 9.5802. Min Range: -9.5802

Max Act: 8.8615. Min Act: -0.1699

Data Index: 258043 (C4 (Web Text))

Max Activating Token Index: 312

Text #11

Max Range: 9.5802. Min Range: -9.5802

Max Act: 8.8459. Min Act: -0.1699

Data Index: 924770 (C4 (Web Text))

Max Activating Token Index: 1006

Text #12

Max Range: 9.5802. Min Range: -9.5802

Max Act: 8.7365. Min Act: -0.1699

Data Index: 1274910 (C4 (Web Text))

Max Activating Token Index: 724

Text #13

Max Range: 9.5802. Min Range: -9.5802

Max Act: 8.7209. Min Act: -0.1699

Data Index: 330669 (C4 (Web Text))

Max Activating Token Index: 636

Text #14

Max Range: 9.5802. Min Range: -9.5802

Max Act: 8.6740. Min Act: -0.1699

Data Index: 273356 (C4 (Web Text))

Max Activating Token Index: 176

Text #15

Max Range: 9.5802. Min Range: -9.5802

Max Act: 8.6740. Min Act: -0.1699

Data Index: 482123 (C4 (Web Text))

Max Activating Token Index: 996

Text #16

Max Range: 9.5802. Min Range: -9.5802

Max Act: 8.5959. Min Act: -0.1699

Data Index: 19470 (C4 (Web Text))

Max Activating Token Index: 597

Text #17

Max Range: 9.5802. Min Range: -9.5802

Max Act: 8.6115. Min Act: -0.1699

Data Index: 474154 (C4 (Web Text))

Max Activating Token Index: 995

Text #18

Max Range: 9.5802. Min Range: -9.5802

Max Act: 8.5334. Min Act: -0.1699

Data Index: 1452045 (Python Code)

Max Activating Token Index: 160

Text #19

Max Range: 9.5802. Min Range: -9.5802

Max Act: 8.4709. Min Act: -0.1699

Data Index: 493489 (C4 (Web Text))

Max Activating Token Index: 261