Model: GELU Model: 1 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 231 in Layer 0

Load this data into an Interactive Neuroscope

See Documentation here

Hooked Transformer Loading: HookedTransformer.from_pretrained('gelu-1l')

Click the toggle to see the full text!

Text #0

Max Range: 4.7529. Min Range: -4.7529

Max Act: 4.7503. Min Act: -0.1699

Data Index: 1348257 (C4 (Web Text))

Max Activating Token Index: 112

Text #1

Max Range: 4.7529. Min Range: -4.7529

Max Act: 4.7529. Min Act: 0.4944

Data Index: 1077861 (C4 (Web Text))

Max Activating Token Index: 523

Text #2

Max Range: 4.7529. Min Range: -4.7529

Max Act: 4.6599. Min Act: -0.0931

Data Index: 1386590 (Python Code)

Max Activating Token Index: 221

Text #3

Max Range: 4.7529. Min Range: -4.7529

Max Act: 4.6393. Min Act: 0.7543

Data Index: 1624003 (Python Code)

Max Activating Token Index: 828

Text #4

Max Range: 4.7529. Min Range: -4.7529

Max Act: 4.5405. Min Act: -0.1679

Data Index: 1237227 (C4 (Web Text))

Max Activating Token Index: 47

Text #5

Max Range: 4.7529. Min Range: -4.7529

Max Act: 4.5614. Min Act: 0.2516

Data Index: 1300552 (C4 (Web Text))

Max Activating Token Index: 12

Text #6

Max Range: 4.7529. Min Range: -4.7529

Max Act: 4.4918. Min Act: 0.6565

Data Index: 881286 (C4 (Web Text))

Max Activating Token Index: 104

Text #7

Max Range: 4.7529. Min Range: -4.7529

Max Act: 4.4685. Min Act: 0.2487

Data Index: 545932 (C4 (Web Text))

Max Activating Token Index: 31

Text #8

Max Range: 4.7529. Min Range: -4.7529

Max Act: 4.4713. Min Act: -0.1697

Data Index: 1038170 (C4 (Web Text))

Max Activating Token Index: 150

Text #9

Max Range: 4.7529. Min Range: -4.7529

Max Act: 4.3892. Min Act: -0.1660

Data Index: 1627685 (Python Code)

Max Activating Token Index: 215

Text #10

Max Range: 4.7529. Min Range: -4.7529

Max Act: 4.3920. Min Act: 0.5837

Data Index: 70505 (C4 (Web Text))

Max Activating Token Index: 901

Text #11

Max Range: 4.7529. Min Range: -4.7529

Max Act: 4.3906. Min Act: -0.1696

Data Index: 225865 (C4 (Web Text))

Max Activating Token Index: 131

Text #12

Max Range: 4.7529. Min Range: -4.7529

Max Act: 4.3961. Min Act: 0.6602

Data Index: 985637 (C4 (Web Text))

Max Activating Token Index: 530

Text #13

Max Range: 4.7529. Min Range: -4.7529

Max Act: 4.3862. Min Act: -0.1693

Data Index: 1345932 (C4 (Web Text))

Max Activating Token Index: 480

Text #14

Max Range: 4.7529. Min Range: -4.7529

Max Act: 4.3774. Min Act: -0.1669

Data Index: 663427 (C4 (Web Text))

Max Activating Token Index: 521

Text #15

Max Range: 4.7529. Min Range: -4.7529

Max Act: 4.3800. Min Act: 0.6548

Data Index: 80050 (C4 (Web Text))

Max Activating Token Index: 31

Text #16

Max Range: 4.7529. Min Range: -4.7529

Max Act: 4.3646. Min Act: -0.1691

Data Index: 948526 (C4 (Web Text))

Max Activating Token Index: 56

Text #17

Max Range: 4.7529. Min Range: -4.7529

Max Act: 4.3481. Min Act: -0.1484

Data Index: 198631 (C4 (Web Text))

Max Activating Token Index: 714

Text #18

Max Range: 4.7529. Min Range: -4.7529

Max Act: 4.3670. Min Act: -0.1457

Data Index: 1250429 (C4 (Web Text))

Max Activating Token Index: 322

Text #19

Max Range: 4.7529. Min Range: -4.7529

Max Act: 4.3976. Min Act: -0.0725

Data Index: 262392 (C4 (Web Text))

Max Activating Token Index: 825