Model: GELU Model: 3 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1753 in Layer 0

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('gelu-3l')

Text #0

Max Range: 3.8201. Min Range: -3.8201

Max Act: 3.8201. Min Act: -0.1700

Data Index: 1170448 (C4 (Web Text))

Max Activating Token Index: 971

Text #1

Max Range: 3.8201. Min Range: -3.8201

Max Act: 3.8045. Min Act: -0.1700

Data Index: 347452 (C4 (Web Text))

Max Activating Token Index: 668

Text #2

Max Range: 3.8201. Min Range: -3.8201

Max Act: 3.7576. Min Act: -0.1700

Data Index: 84022 (C4 (Web Text))

Max Activating Token Index: 919

Text #3

Max Range: 3.8201. Min Range: -3.8201

Max Act: 3.7576. Min Act: -0.1700

Data Index: 805346 (C4 (Web Text))

Max Activating Token Index: 830

Text #4

Max Range: 3.8201. Min Range: -3.8201

Max Act: 3.7106. Min Act: -0.1700

Data Index: 540676 (C4 (Web Text))

Max Activating Token Index: 60

Text #5

Max Range: 3.8201. Min Range: -3.8201

Max Act: 3.7106. Min Act: -0.1700

Data Index: 1313798 (C4 (Web Text))

Max Activating Token Index: 535

Text #6

Max Range: 3.8201. Min Range: -3.8201

Max Act: 3.6793. Min Act: -0.1700

Data Index: 568083 (C4 (Web Text))

Max Activating Token Index: 558

Text #7

Max Range: 3.8201. Min Range: -3.8201

Max Act: 3.6637. Min Act: -0.1700

Data Index: 965933 (C4 (Web Text))

Max Activating Token Index: 575

Text #8

Max Range: 3.8201. Min Range: -3.8201

Max Act: 3.6793. Min Act: -0.1700

Data Index: 1226141 (C4 (Web Text))

Max Activating Token Index: 355

Text #9

Max Range: 3.8201. Min Range: -3.8201

Max Act: 3.6324. Min Act: -0.1700

Data Index: 1044020 (C4 (Web Text))

Max Activating Token Index: 21

Text #10

Max Range: 3.8201. Min Range: -3.8201

Max Act: 3.6324. Min Act: -0.1700

Data Index: 1032421 (C4 (Web Text))

Max Activating Token Index: 790

Text #11

Max Range: 3.8201. Min Range: -3.8201

Max Act: 3.6167. Min Act: -0.1700

Data Index: 1220019 (C4 (Web Text))

Max Activating Token Index: 286

Text #12

Max Range: 3.8201. Min Range: -3.8201

Max Act: 3.6167. Min Act: -0.1700

Data Index: 417545 (C4 (Web Text))

Max Activating Token Index: 175

Text #13

Max Range: 3.8201. Min Range: -3.8201

Max Act: 3.5854. Min Act: -0.1700

Data Index: 215299 (C4 (Web Text))

Max Activating Token Index: 119

Text #14

Max Range: 3.8201. Min Range: -3.8201

Max Act: 3.5854. Min Act: -0.1700

Data Index: 789724 (C4 (Web Text))

Max Activating Token Index: 221

Text #15

Max Range: 3.8201. Min Range: -3.8201

Max Act: 3.5698. Min Act: -0.1700

Data Index: 855869 (C4 (Web Text))

Max Activating Token Index: 13

Text #16

Max Range: 3.8201. Min Range: -3.8201

Max Act: 3.5541. Min Act: -0.1700

Data Index: 938990 (C4 (Web Text))

Max Activating Token Index: 3

Text #17

Max Range: 3.8201. Min Range: -3.8201

Max Act: 3.5698. Min Act: -0.1700

Data Index: 424108 (C4 (Web Text))

Max Activating Token Index: 881

Text #18

Max Range: 3.8201. Min Range: -3.8201

Max Act: 3.5541. Min Act: -0.1700

Data Index: 1237493 (C4 (Web Text))

Max Activating Token Index: 829

Text #19

Max Range: 3.8201. Min Range: -3.8201

Max Act: 3.5541. Min Act: -0.1700

Data Index: 950599 (C4 (Web Text))

Max Activating Token Index: 1002