Model: GELU Model: 3 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1107 in Layer 0

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('gelu-3l')

Text #0

Max Range: 3.0028. Min Range: -3.0028

Max Act: 3.0028. Min Act: -0.1700

Data Index: 694484 (C4 (Web Text))

Max Activating Token Index: 506

Text #1

Max Range: 3.0028. Min Range: -3.0028

Max Act: 2.9870. Min Act: -0.1700

Data Index: 989410 (C4 (Web Text))

Max Activating Token Index: 101

Text #2

Max Range: 3.0028. Min Range: -3.0028

Max Act: 2.9554. Min Act: -0.1700

Data Index: 790905 (C4 (Web Text))

Max Activating Token Index: 272

Text #3

Max Range: 3.0028. Min Range: -3.0028

Max Act: 2.9395. Min Act: -0.1700

Data Index: 740782 (C4 (Web Text))

Max Activating Token Index: 374

Text #4

Max Range: 3.0028. Min Range: -3.0028

Max Act: 2.8920. Min Act: -0.1700

Data Index: 1020511 (C4 (Web Text))

Max Activating Token Index: 416

Text #5

Max Range: 3.0028. Min Range: -3.0028

Max Act: 2.8761. Min Act: -0.1700

Data Index: 462709 (C4 (Web Text))

Max Activating Token Index: 27

Text #6

Max Range: 3.0028. Min Range: -3.0028

Max Act: 2.8761. Min Act: -0.1700

Data Index: 797362 (C4 (Web Text))

Max Activating Token Index: 1009

Text #7

Max Range: 3.0028. Min Range: -3.0028

Max Act: 2.8602. Min Act: -0.1700

Data Index: 1241273 (C4 (Web Text))

Max Activating Token Index: 351

Text #8

Max Range: 3.0028. Min Range: -3.0028

Max Act: 2.8602. Min Act: -0.1700

Data Index: 722278 (C4 (Web Text))

Max Activating Token Index: 459

Text #9

Max Range: 3.0028. Min Range: -3.0028

Max Act: 2.8602. Min Act: -0.1700

Data Index: 173191 (C4 (Web Text))

Max Activating Token Index: 644

Text #10

Max Range: 3.0028. Min Range: -3.0028

Max Act: 2.8443. Min Act: -0.1700

Data Index: 720483 (C4 (Web Text))

Max Activating Token Index: 796

Text #11

Max Range: 3.0028. Min Range: -3.0028

Max Act: 2.8602. Min Act: -0.1700

Data Index: 352787 (C4 (Web Text))

Max Activating Token Index: 421

Text #12

Max Range: 3.0028. Min Range: -3.0028

Max Act: 2.8443. Min Act: -0.1700

Data Index: 302523 (C4 (Web Text))

Max Activating Token Index: 500

Text #13

Max Range: 3.0028. Min Range: -3.0028

Max Act: 2.8125. Min Act: -0.1700

Data Index: 338968 (C4 (Web Text))

Max Activating Token Index: 171

Text #14

Max Range: 3.0028. Min Range: -3.0028

Max Act: 2.8284. Min Act: -0.1700

Data Index: 801325 (C4 (Web Text))

Max Activating Token Index: 400

Text #15

Max Range: 3.0028. Min Range: -3.0028

Max Act: 2.8125. Min Act: -0.1700

Data Index: 115167 (C4 (Web Text))

Max Activating Token Index: 993

Text #16

Max Range: 3.0028. Min Range: -3.0028

Max Act: 2.7966. Min Act: -0.1700

Data Index: 445248 (C4 (Web Text))

Max Activating Token Index: 616

Text #17

Max Range: 3.0028. Min Range: -3.0028

Max Act: 2.7966. Min Act: -0.1700

Data Index: 1220757 (C4 (Web Text))

Max Activating Token Index: 341

Text #18

Max Range: 3.0028. Min Range: -3.0028

Max Act: 2.7966. Min Act: -0.1700

Data Index: 1037845 (C4 (Web Text))

Max Activating Token Index: 429

Text #19

Max Range: 3.0028. Min Range: -3.0028

Max Act: 2.7807. Min Act: -0.1700

Data Index: 772758 (C4 (Web Text))

Max Activating Token Index: 6