Model: GELU Model: 4 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 786 in Layer 3

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('gelu-4l')

Text #0

Max Range: 14.7669. Min Range: -14.7669

Max Act: 14.7669. Min Act: -0.1700

Data Index: 303013 (C4 (Web Text))

Max Activating Token Index: 179

Text #1

Max Range: 14.7669. Min Range: -14.7669

Max Act: 14.3294. Min Act: -0.1700

Data Index: 1289653 (C4 (Web Text))

Max Activating Token Index: 1016

Text #2

Max Range: 14.7669. Min Range: -14.7669

Max Act: 14.2044. Min Act: -0.1700

Data Index: 497282 (C4 (Web Text))

Max Activating Token Index: 999

Text #3

Max Range: 14.7669. Min Range: -14.7669

Max Act: 14.1419. Min Act: -0.1700

Data Index: 468917 (C4 (Web Text))

Max Activating Token Index: 188

Text #4

Max Range: 14.7669. Min Range: -14.7669

Max Act: 14.0794. Min Act: -0.1700

Data Index: 198944 (C4 (Web Text))

Max Activating Token Index: 176

Text #5

Max Range: 14.7669. Min Range: -14.7669

Max Act: 14.1419. Min Act: -0.1700

Data Index: 893302 (C4 (Web Text))

Max Activating Token Index: 355

Text #6

Max Range: 14.7669. Min Range: -14.7669

Max Act: 14.0794. Min Act: -0.1700

Data Index: 647760 (C4 (Web Text))

Max Activating Token Index: 1009

Text #7

Max Range: 14.7669. Min Range: -14.7669

Max Act: 14.0169. Min Act: -0.1700

Data Index: 1005291 (C4 (Web Text))

Max Activating Token Index: 371

Text #8

Max Range: 14.7669. Min Range: -14.7669

Max Act: 14.0169. Min Act: -0.1699

Data Index: 605785 (C4 (Web Text))

Max Activating Token Index: 351

Text #9

Max Range: 14.7669. Min Range: -14.7669

Max Act: 14.0169. Min Act: -0.1700

Data Index: 477436 (C4 (Web Text))

Max Activating Token Index: 334

Text #10

Max Range: 14.7669. Min Range: -14.7669

Max Act: 13.8294. Min Act: -0.1700

Data Index: 76439 (C4 (Web Text))

Max Activating Token Index: 516

Text #11

Max Range: 14.7669. Min Range: -14.7669

Max Act: 13.8919. Min Act: -0.1700

Data Index: 1102496 (C4 (Web Text))

Max Activating Token Index: 374

Text #12

Max Range: 14.7669. Min Range: -14.7669

Max Act: 13.8919. Min Act: -0.1700

Data Index: 96443 (C4 (Web Text))

Max Activating Token Index: 524

Text #13

Max Range: 14.7669. Min Range: -14.7669

Max Act: 13.8294. Min Act: -0.1700

Data Index: 239973 (C4 (Web Text))

Max Activating Token Index: 272

Text #14

Max Range: 14.7669. Min Range: -14.7669

Max Act: 13.7669. Min Act: -0.1700

Data Index: 1118721 (C4 (Web Text))

Max Activating Token Index: 774

Text #15

Max Range: 14.7669. Min Range: -14.7669

Max Act: 13.6419. Min Act: -0.1700

Data Index: 63189 (C4 (Web Text))

Max Activating Token Index: 815

Text #16

Max Range: 14.7669. Min Range: -14.7669

Max Act: 13.7044. Min Act: -0.1700

Data Index: 790390 (C4 (Web Text))

Max Activating Token Index: 979

Text #17

Max Range: 14.7669. Min Range: -14.7669

Max Act: 13.7669. Min Act: -0.1700

Data Index: 963974 (C4 (Web Text))

Max Activating Token Index: 916

Text #18

Max Range: 14.7669. Min Range: -14.7669

Max Act: 13.7669. Min Act: -0.1700

Data Index: 979854 (C4 (Web Text))

Max Activating Token Index: 247

Text #19

Max Range: 14.7669. Min Range: -14.7669

Max Act: 13.8294. Min Act: -0.1700

Data Index: 881756 (C4 (Web Text))

Max Activating Token Index: 179