Model: GELU Model: 3 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 346 in Layer 1

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('gelu-3l')

Text #0

Max Range: 5.5345. Min Range: -5.5345

Max Act: 5.5345. Min Act: -0.1700

Data Index: 650930 (C4 (Web Text))

Max Activating Token Index: 1010

Text #1

Max Range: 5.5345. Min Range: -5.5345

Max Act: 5.4408. Min Act: -0.1700

Data Index: 553527 (C4 (Web Text))

Max Activating Token Index: 239

Text #2

Max Range: 5.5345. Min Range: -5.5345

Max Act: 5.4720. Min Act: -0.1700

Data Index: 426547 (C4 (Web Text))

Max Activating Token Index: 761

Text #3

Max Range: 5.5345. Min Range: -5.5345

Max Act: 5.4408. Min Act: -0.1700

Data Index: 547455 (C4 (Web Text))

Max Activating Token Index: 499

Text #4

Max Range: 5.5345. Min Range: -5.5345

Max Act: 5.2533. Min Act: -0.1700

Data Index: 355060 (C4 (Web Text))

Max Activating Token Index: 188

Text #5

Max Range: 5.5345. Min Range: -5.5345

Max Act: 5.4095. Min Act: -0.1700

Data Index: 1309220 (C4 (Web Text))

Max Activating Token Index: 121

Text #6

Max Range: 5.5345. Min Range: -5.5345

Max Act: 5.3470. Min Act: -0.1700

Data Index: 1081938 (C4 (Web Text))

Max Activating Token Index: 518

Text #7

Max Range: 5.5345. Min Range: -5.5345

Max Act: 5.3470. Min Act: -0.1700

Data Index: 155702 (C4 (Web Text))

Max Activating Token Index: 481

Text #8

Max Range: 5.5345. Min Range: -5.5345

Max Act: 5.2845. Min Act: -0.1700

Data Index: 146882 (C4 (Web Text))

Max Activating Token Index: 648

Text #9

Max Range: 5.5345. Min Range: -5.5345

Max Act: 5.2845. Min Act: -0.1700

Data Index: 581499 (C4 (Web Text))

Max Activating Token Index: 810

Text #10

Max Range: 5.5345. Min Range: -5.5345

Max Act: 5.2533. Min Act: -0.1700

Data Index: 166392 (C4 (Web Text))

Max Activating Token Index: 537

Text #11

Max Range: 5.5345. Min Range: -5.5345

Max Act: 5.3158. Min Act: -0.1700

Data Index: 170951 (C4 (Web Text))

Max Activating Token Index: 308

Text #12

Max Range: 5.5345. Min Range: -5.5345

Max Act: 5.3158. Min Act: -0.1700

Data Index: 485344 (C4 (Web Text))

Max Activating Token Index: 409

Text #13

Max Range: 5.5345. Min Range: -5.5345

Max Act: 5.3470. Min Act: -0.1700

Data Index: 906124 (C4 (Web Text))

Max Activating Token Index: 737

Text #14

Max Range: 5.5345. Min Range: -5.5345

Max Act: 5.2220. Min Act: -0.1700

Data Index: 238287 (C4 (Web Text))

Max Activating Token Index: 505

Text #15

Max Range: 5.5345. Min Range: -5.5345

Max Act: 5.3158. Min Act: -0.1700

Data Index: 841259 (C4 (Web Text))

Max Activating Token Index: 791

Text #16

Max Range: 5.5345. Min Range: -5.5345

Max Act: 5.2533. Min Act: -0.1700

Data Index: 1267996 (C4 (Web Text))

Max Activating Token Index: 670

Text #17

Max Range: 5.5345. Min Range: -5.5345

Max Act: 5.2533. Min Act: -0.1700

Data Index: 656455 (C4 (Web Text))

Max Activating Token Index: 519

Text #18

Max Range: 5.5345. Min Range: -5.5345

Max Act: 5.2845. Min Act: -0.1700

Data Index: 242211 (C4 (Web Text))

Max Activating Token Index: 706

Text #19

Max Range: 5.5345. Min Range: -5.5345

Max Act: 5.2533. Min Act: -0.1700

Data Index: 880837 (C4 (Web Text))

Max Activating Token Index: 341