Model: GELU Model: 3 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1255 in Layer 1

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('gelu-3l')

Text #0

Max Range: 5.5016. Min Range: -5.5016

Max Act: 5.5016. Min Act: -0.1700

Data Index: 1254114 (C4 (Web Text))

Max Activating Token Index: 451

Text #1

Max Range: 5.5016. Min Range: -5.5016

Max Act: 5.3766. Min Act: -0.1700

Data Index: 264692 (C4 (Web Text))

Max Activating Token Index: 742

Text #2

Max Range: 5.5016. Min Range: -5.5016

Max Act: 5.3453. Min Act: -0.1700

Data Index: 328156 (C4 (Web Text))

Max Activating Token Index: 544

Text #3

Max Range: 5.5016. Min Range: -5.5016

Max Act: 5.3141. Min Act: -0.1700

Data Index: 398063 (C4 (Web Text))

Max Activating Token Index: 444

Text #4

Max Range: 5.5016. Min Range: -5.5016

Max Act: 5.3141. Min Act: -0.1699

Data Index: 1349462 (C4 (Web Text))

Max Activating Token Index: 125

Text #5

Max Range: 5.5016. Min Range: -5.5016

Max Act: 5.2828. Min Act: -0.1700

Data Index: 996080 (C4 (Web Text))

Max Activating Token Index: 215

Text #6

Max Range: 5.5016. Min Range: -5.5016

Max Act: 5.2828. Min Act: -0.1700

Data Index: 738533 (C4 (Web Text))

Max Activating Token Index: 334

Text #7

Max Range: 5.5016. Min Range: -5.5016

Max Act: 5.2828. Min Act: -0.1700

Data Index: 1245779 (C4 (Web Text))

Max Activating Token Index: 159

Text #8

Max Range: 5.5016. Min Range: -5.5016

Max Act: 5.2203. Min Act: -0.1700

Data Index: 224283 (C4 (Web Text))

Max Activating Token Index: 106

Text #9

Max Range: 5.5016. Min Range: -5.5016

Max Act: 5.2203. Min Act: -0.1700

Data Index: 821992 (C4 (Web Text))

Max Activating Token Index: 428

Text #10

Max Range: 5.5016. Min Range: -5.5016

Max Act: 5.2203. Min Act: -0.1700

Data Index: 863106 (C4 (Web Text))

Max Activating Token Index: 471

Text #11

Max Range: 5.5016. Min Range: -5.5016

Max Act: 5.2516. Min Act: -0.1700

Data Index: 485945 (C4 (Web Text))

Max Activating Token Index: 177

Text #12

Max Range: 5.5016. Min Range: -5.5016

Max Act: 5.2203. Min Act: -0.1700

Data Index: 216105 (C4 (Web Text))

Max Activating Token Index: 729

Text #13

Max Range: 5.5016. Min Range: -5.5016

Max Act: 5.2203. Min Act: -0.1700

Data Index: 283642 (C4 (Web Text))

Max Activating Token Index: 351

Text #14

Max Range: 5.5016. Min Range: -5.5016

Max Act: 5.2203. Min Act: -0.1700

Data Index: 1108150 (C4 (Web Text))

Max Activating Token Index: 241

Text #15

Max Range: 5.5016. Min Range: -5.5016

Max Act: 5.2203. Min Act: -0.1700

Data Index: 372458 (C4 (Web Text))

Max Activating Token Index: 441

Text #16

Max Range: 5.5016. Min Range: -5.5016

Max Act: 5.1578. Min Act: -0.1700

Data Index: 928205 (C4 (Web Text))

Max Activating Token Index: 702

Text #17

Max Range: 5.5016. Min Range: -5.5016

Max Act: 5.1891. Min Act: -0.1699

Data Index: 104092 (C4 (Web Text))

Max Activating Token Index: 335

Text #18

Max Range: 5.5016. Min Range: -5.5016

Max Act: 5.1891. Min Act: -0.1700

Data Index: 685979 (C4 (Web Text))

Max Activating Token Index: 494

Text #19

Max Range: 5.5016. Min Range: -5.5016

Max Act: 5.1578. Min Act: -0.1700

Data Index: 1149289 (C4 (Web Text))

Max Activating Token Index: 550