Model: GELU Model: 2 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1268 in Layer 1

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('gelu-2l')

Text #0

Max Range: 5.3003. Min Range: -5.3003

Max Act: 5.2690. Min Act: -0.1700

Data Index: 1352587 (C4 (Web Text))

Max Activating Token Index: 525

Text #1

Max Range: 5.3003. Min Range: -5.3003

Max Act: 5.1440. Min Act: -0.1700

Data Index: 1082369 (C4 (Web Text))

Max Activating Token Index: 450

Text #2

Max Range: 5.3003. Min Range: -5.3003

Max Act: 5.3003. Min Act: -0.1700

Data Index: 1147296 (C4 (Web Text))

Max Activating Token Index: 555

Text #3

Max Range: 5.3003. Min Range: -5.3003

Max Act: 5.2378. Min Act: -0.1700

Data Index: 264946 (C4 (Web Text))

Max Activating Token Index: 610

Text #4

Max Range: 5.3003. Min Range: -5.3003

Max Act: 5.3003. Min Act: -0.1700

Data Index: 1057113 (C4 (Web Text))

Max Activating Token Index: 555

Text #5

Max Range: 5.3003. Min Range: -5.3003

Max Act: 5.0815. Min Act: -0.1700

Data Index: 964138 (C4 (Web Text))

Max Activating Token Index: 881

Text #6

Max Range: 5.3003. Min Range: -5.3003

Max Act: 5.1128. Min Act: -0.1700

Data Index: 330726 (C4 (Web Text))

Max Activating Token Index: 499

Text #7

Max Range: 5.3003. Min Range: -5.3003

Max Act: 5.0815. Min Act: -0.1700

Data Index: 402952 (C4 (Web Text))

Max Activating Token Index: 940

Text #8

Max Range: 5.3003. Min Range: -5.3003

Max Act: 5.1440. Min Act: -0.1700

Data Index: 858853 (C4 (Web Text))

Max Activating Token Index: 589

Text #9

Max Range: 5.3003. Min Range: -5.3003

Max Act: 5.0815. Min Act: -0.1700

Data Index: 1106630 (C4 (Web Text))

Max Activating Token Index: 872

Text #10

Max Range: 5.3003. Min Range: -5.3003

Max Act: 5.2065. Min Act: -0.1700

Data Index: 290089 (C4 (Web Text))

Max Activating Token Index: 586

Text #11

Max Range: 5.3003. Min Range: -5.3003

Max Act: 5.2065. Min Act: -0.1700

Data Index: 1002570 (C4 (Web Text))

Max Activating Token Index: 849

Text #12

Max Range: 5.3003. Min Range: -5.3003

Max Act: 5.0190. Min Act: -0.1700

Data Index: 635408 (C4 (Web Text))

Max Activating Token Index: 653

Text #13

Max Range: 5.3003. Min Range: -5.3003

Max Act: 5.0190. Min Act: -0.1700

Data Index: 301872 (C4 (Web Text))

Max Activating Token Index: 402

Text #14

Max Range: 5.3003. Min Range: -5.3003

Max Act: 5.1128. Min Act: -0.1700

Data Index: 85922 (C4 (Web Text))

Max Activating Token Index: 443

Text #15

Max Range: 5.3003. Min Range: -5.3003

Max Act: 4.9253. Min Act: -0.1700

Data Index: 926523 (C4 (Web Text))

Max Activating Token Index: 259

Text #16

Max Range: 5.3003. Min Range: -5.3003

Max Act: 5.0190. Min Act: -0.1700

Data Index: 189401 (C4 (Web Text))

Max Activating Token Index: 479

Text #17

Max Range: 5.3003. Min Range: -5.3003

Max Act: 5.0503. Min Act: -0.1700

Data Index: 715388 (C4 (Web Text))

Max Activating Token Index: 883

Text #18

Max Range: 5.3003. Min Range: -5.3003

Max Act: 4.9253. Min Act: -0.1700

Data Index: 43952 (C4 (Web Text))

Max Activating Token Index: 976

Text #19

Max Range: 5.3003. Min Range: -5.3003

Max Act: 4.9565. Min Act: -0.1700

Data Index: 212813 (C4 (Web Text))

Max Activating Token Index: 412