Model: GELU Model: 2 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 261 in Layer 1

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('gelu-2l')

Text #0

Max Range: 5.4054. Min Range: -5.4054

Max Act: 5.4054. Min Act: -0.1700

Data Index: 523070 (C4 (Web Text))

Max Activating Token Index: 355

Text #1

Max Range: 5.4054. Min Range: -5.4054

Max Act: 5.4054. Min Act: -0.1700

Data Index: 37548 (C4 (Web Text))

Max Activating Token Index: 642

Text #2

Max Range: 5.4054. Min Range: -5.4054

Max Act: 5.3116. Min Act: -0.1700

Data Index: 605761 (C4 (Web Text))

Max Activating Token Index: 373

Text #3

Max Range: 5.4054. Min Range: -5.4054

Max Act: 5.3429. Min Act: -0.1700

Data Index: 397484 (C4 (Web Text))

Max Activating Token Index: 439

Text #4

Max Range: 5.4054. Min Range: -5.4054

Max Act: 5.3116. Min Act: -0.1700

Data Index: 644039 (C4 (Web Text))

Max Activating Token Index: 994

Text #5

Max Range: 5.4054. Min Range: -5.4054

Max Act: 5.2804. Min Act: -0.1700

Data Index: 476385 (C4 (Web Text))

Max Activating Token Index: 523

Text #6

Max Range: 5.4054. Min Range: -5.4054

Max Act: 5.2804. Min Act: -0.1700

Data Index: 1133827 (C4 (Web Text))

Max Activating Token Index: 182

Text #7

Max Range: 5.4054. Min Range: -5.4054

Max Act: 5.2804. Min Act: -0.1700

Data Index: 533871 (C4 (Web Text))

Max Activating Token Index: 489

Text #8

Max Range: 5.4054. Min Range: -5.4054

Max Act: 5.2804. Min Act: -0.1700

Data Index: 797325 (C4 (Web Text))

Max Activating Token Index: 116

Text #9

Max Range: 5.4054. Min Range: -5.4054

Max Act: 5.3116. Min Act: -0.1700

Data Index: 674312 (C4 (Web Text))

Max Activating Token Index: 957

Text #10

Max Range: 5.4054. Min Range: -5.4054

Max Act: 5.2804. Min Act: -0.1700

Data Index: 638771 (C4 (Web Text))

Max Activating Token Index: 87

Text #11

Max Range: 5.4054. Min Range: -5.4054

Max Act: 5.2804. Min Act: -0.1700

Data Index: 1359593 (C4 (Web Text))

Max Activating Token Index: 781

Text #12

Max Range: 5.4054. Min Range: -5.4054

Max Act: 5.1866. Min Act: -0.1700

Data Index: 277460 (C4 (Web Text))

Max Activating Token Index: 482

Text #13

Max Range: 5.4054. Min Range: -5.4054

Max Act: 5.2804. Min Act: -0.1700

Data Index: 70380 (C4 (Web Text))

Max Activating Token Index: 396

Text #14

Max Range: 5.4054. Min Range: -5.4054

Max Act: 5.2179. Min Act: -0.1700

Data Index: 1343077 (C4 (Web Text))

Max Activating Token Index: 762

Text #15

Max Range: 5.4054. Min Range: -5.4054

Max Act: 5.2491. Min Act: -0.1700

Data Index: 1111881 (C4 (Web Text))

Max Activating Token Index: 733

Text #16

Max Range: 5.4054. Min Range: -5.4054

Max Act: 5.2491. Min Act: -0.1700

Data Index: 969195 (C4 (Web Text))

Max Activating Token Index: 121

Text #17

Max Range: 5.4054. Min Range: -5.4054

Max Act: 5.2179. Min Act: -0.1700

Data Index: 114918 (C4 (Web Text))

Max Activating Token Index: 971

Text #18

Max Range: 5.4054. Min Range: -5.4054

Max Act: 5.2179. Min Act: -0.1700

Data Index: 39708 (C4 (Web Text))

Max Activating Token Index: 1001

Text #19

Max Range: 5.4054. Min Range: -5.4054

Max Act: 5.2179. Min Act: -0.1700

Data Index: 417741 (C4 (Web Text))

Max Activating Token Index: 131