Model: GELU Model: 4 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 76 in Layer 1

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('gelu-4l')

Text #0

Max Range: 5.5183. Min Range: -5.5183

Max Act: 5.5183. Min Act: -0.1700

Data Index: 666332 (C4 (Web Text))

Max Activating Token Index: 84

Text #1

Max Range: 5.5183. Min Range: -5.5183

Max Act: 5.3621. Min Act: -0.1700

Data Index: 259202 (C4 (Web Text))

Max Activating Token Index: 283

Text #2

Max Range: 5.5183. Min Range: -5.5183

Max Act: 5.2996. Min Act: -0.1700

Data Index: 211884 (C4 (Web Text))

Max Activating Token Index: 143

Text #3

Max Range: 5.5183. Min Range: -5.5183

Max Act: 5.2683. Min Act: -0.1700

Data Index: 45153 (C4 (Web Text))

Max Activating Token Index: 355

Text #4

Max Range: 5.5183. Min Range: -5.5183

Max Act: 5.2996. Min Act: -0.1700

Data Index: 125924 (C4 (Web Text))

Max Activating Token Index: 65

Text #5

Max Range: 5.5183. Min Range: -5.5183

Max Act: 5.2371. Min Act: -0.1700

Data Index: 757290 (C4 (Web Text))

Max Activating Token Index: 120

Text #6

Max Range: 5.5183. Min Range: -5.5183

Max Act: 5.2058. Min Act: -0.1700

Data Index: 1310448 (C4 (Web Text))

Max Activating Token Index: 277

Text #7

Max Range: 5.5183. Min Range: -5.5183

Max Act: 5.2371. Min Act: -0.1700

Data Index: 722151 (C4 (Web Text))

Max Activating Token Index: 452

Text #8

Max Range: 5.5183. Min Range: -5.5183

Max Act: 5.1746. Min Act: -0.1700

Data Index: 683500 (C4 (Web Text))

Max Activating Token Index: 200

Text #9

Max Range: 5.5183. Min Range: -5.5183

Max Act: 5.2058. Min Act: -0.1700

Data Index: 513639 (C4 (Web Text))

Max Activating Token Index: 51

Text #10

Max Range: 5.5183. Min Range: -5.5183

Max Act: 5.1746. Min Act: -0.1700

Data Index: 312044 (C4 (Web Text))

Max Activating Token Index: 66

Text #11

Max Range: 5.5183. Min Range: -5.5183

Max Act: 5.1746. Min Act: -0.1700

Data Index: 184197 (C4 (Web Text))

Max Activating Token Index: 430

Text #12

Max Range: 5.5183. Min Range: -5.5183

Max Act: 5.1433. Min Act: -0.1700

Data Index: 1178534 (C4 (Web Text))

Max Activating Token Index: 398

Text #13

Max Range: 5.5183. Min Range: -5.5183

Max Act: 5.1433. Min Act: -0.1700

Data Index: 193597 (C4 (Web Text))

Max Activating Token Index: 315

Text #14

Max Range: 5.5183. Min Range: -5.5183

Max Act: 5.1433. Min Act: -0.1700

Data Index: 411940 (C4 (Web Text))

Max Activating Token Index: 53

Text #15

Max Range: 5.5183. Min Range: -5.5183

Max Act: 5.1433. Min Act: -0.1700

Data Index: 795363 (C4 (Web Text))

Max Activating Token Index: 53

Text #16

Max Range: 5.5183. Min Range: -5.5183

Max Act: 5.1121. Min Act: -0.1700

Data Index: 245155 (C4 (Web Text))

Max Activating Token Index: 472

Text #17

Max Range: 5.5183. Min Range: -5.5183

Max Act: 5.1433. Min Act: -0.1700

Data Index: 724746 (C4 (Web Text))

Max Activating Token Index: 446

Text #18

Max Range: 5.5183. Min Range: -5.5183

Max Act: 5.1121. Min Act: -0.1700

Data Index: 786260 (C4 (Web Text))

Max Activating Token Index: 176

Text #19

Max Range: 5.5183. Min Range: -5.5183

Max Act: 5.1433. Min Act: -0.1700

Data Index: 712915 (C4 (Web Text))

Max Activating Token Index: 806