Model: GELU Model: 3 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 908 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('gelu-3l')

Text #0

Max Range: 5.7372. Min Range: -5.7372

Max Act: 5.7372. Min Act: -0.1700

Data Index: 17762 (C4 (Web Text))

Max Activating Token Index: 185

Text #1

Max Range: 5.7372. Min Range: -5.7372

Max Act: 5.6825. Min Act: -0.1700

Data Index: 13258 (C4 (Web Text))

Max Activating Token Index: 256

Text #2

Max Range: 5.7372. Min Range: -5.7372

Max Act: 5.3290. Min Act: -0.1700

Data Index: 1088384 (C4 (Web Text))

Max Activating Token Index: 482

Text #3

Max Range: 5.7372. Min Range: -5.7372

Max Act: 5.2870. Min Act: -0.1700

Data Index: 165277 (C4 (Web Text))

Max Activating Token Index: 640

Text #4

Max Range: 5.7372. Min Range: -5.7372

Max Act: 5.2098. Min Act: -0.1700

Data Index: 160022 (C4 (Web Text))

Max Activating Token Index: 833

Text #5

Max Range: 5.7372. Min Range: -5.7372

Max Act: 5.1879. Min Act: -0.1700

Data Index: 721626 (C4 (Web Text))

Max Activating Token Index: 675

Text #6

Max Range: 5.7372. Min Range: -5.7372

Max Act: 5.1715. Min Act: -0.1700

Data Index: 352768 (C4 (Web Text))

Max Activating Token Index: 903

Text #7

Max Range: 5.7372. Min Range: -5.7372

Max Act: 5.0919. Min Act: -0.1700

Data Index: 1370504 (Python Code)

Max Activating Token Index: 616

Text #8

Max Range: 5.7372. Min Range: -5.7372

Max Act: 5.0619. Min Act: -0.1700

Data Index: 1477734 (Python Code)

Max Activating Token Index: 195

Text #9

Max Range: 5.7372. Min Range: -5.7372

Max Act: 5.0812. Min Act: -0.1700

Data Index: 923430 (C4 (Web Text))

Max Activating Token Index: 113

Text #10

Max Range: 5.7372. Min Range: -5.7372

Max Act: 5.0389. Min Act: -0.1700

Data Index: 958483 (C4 (Web Text))

Max Activating Token Index: 337

Text #11

Max Range: 5.7372. Min Range: -5.7372

Max Act: 4.9901. Min Act: -0.1700

Data Index: 923244 (C4 (Web Text))

Max Activating Token Index: 808

Text #12

Max Range: 5.7372. Min Range: -5.7372

Max Act: 4.9882. Min Act: -0.1700

Data Index: 322588 (C4 (Web Text))

Max Activating Token Index: 328

Text #13

Max Range: 5.7372. Min Range: -5.7372

Max Act: 5.0116. Min Act: -0.1700

Data Index: 1039753 (C4 (Web Text))

Max Activating Token Index: 697

Text #14

Max Range: 5.7372. Min Range: -5.7372

Max Act: 5.0214. Min Act: -0.1700

Data Index: 1401594 (Python Code)

Max Activating Token Index: 665

Text #15

Max Range: 5.7372. Min Range: -5.7372

Max Act: 4.9159. Min Act: -0.1700

Data Index: 758668 (C4 (Web Text))

Max Activating Token Index: 882

Text #16

Max Range: 5.7372. Min Range: -5.7372

Max Act: 4.9403. Min Act: -0.1700

Data Index: 537155 (C4 (Web Text))

Max Activating Token Index: 770

Text #17

Max Range: 5.7372. Min Range: -5.7372

Max Act: 4.9891. Min Act: -0.1700

Data Index: 380048 (C4 (Web Text))

Max Activating Token Index: 957

Text #18

Max Range: 5.7372. Min Range: -5.7372

Max Act: 4.9462. Min Act: -0.1700

Data Index: 273861 (C4 (Web Text))

Max Activating Token Index: 991

Text #19

Max Range: 5.7372. Min Range: -5.7372

Max Act: 4.9227. Min Act: -0.1700

Data Index: 262340 (C4 (Web Text))

Max Activating Token Index: 818