Model: GELU Model: 3 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 395 in Layer 0

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('gelu-3l')

Text #0

Max Range: 3.5911. Min Range: -3.5911

Max Act: 3.5911. Min Act: -0.1700

Data Index: 1175451 (C4 (Web Text))

Max Activating Token Index: 337

Text #1

Max Range: 3.5911. Min Range: -3.5911

Max Act: 3.5441. Min Act: -0.1700

Data Index: 328269 (C4 (Web Text))

Max Activating Token Index: 591

Text #2

Max Range: 3.5911. Min Range: -3.5911

Max Act: 3.5128. Min Act: -0.1700

Data Index: 1319168 (C4 (Web Text))

Max Activating Token Index: 743

Text #3

Max Range: 3.5911. Min Range: -3.5911

Max Act: 3.4971. Min Act: -0.1700

Data Index: 994842 (C4 (Web Text))

Max Activating Token Index: 893

Text #4

Max Range: 3.5911. Min Range: -3.5911

Max Act: 3.4814. Min Act: -0.1700

Data Index: 528866 (C4 (Web Text))

Max Activating Token Index: 937

Text #5

Max Range: 3.5911. Min Range: -3.5911

Max Act: 3.4657. Min Act: -0.1700

Data Index: 956175 (C4 (Web Text))

Max Activating Token Index: 357

Text #6

Max Range: 3.5911. Min Range: -3.5911

Max Act: 3.4501. Min Act: -0.1700

Data Index: 190733 (C4 (Web Text))

Max Activating Token Index: 159

Text #7

Max Range: 3.5911. Min Range: -3.5911

Max Act: 3.4657. Min Act: -0.1700

Data Index: 4010 (C4 (Web Text))

Max Activating Token Index: 165

Text #8

Max Range: 3.5911. Min Range: -3.5911

Max Act: 3.4814. Min Act: -0.1700

Data Index: 250587 (C4 (Web Text))

Max Activating Token Index: 789

Text #9

Max Range: 3.5911. Min Range: -3.5911

Max Act: 3.4657. Min Act: -0.1700

Data Index: 1231986 (C4 (Web Text))

Max Activating Token Index: 700

Text #10

Max Range: 3.5911. Min Range: -3.5911

Max Act: 3.4657. Min Act: -0.1700

Data Index: 134657 (C4 (Web Text))

Max Activating Token Index: 374

Text #11

Max Range: 3.5911. Min Range: -3.5911

Max Act: 3.4501. Min Act: -0.1700

Data Index: 176989 (C4 (Web Text))

Max Activating Token Index: 257

Text #12

Max Range: 3.5911. Min Range: -3.5911

Max Act: 3.4657. Min Act: -0.1700

Data Index: 807052 (C4 (Web Text))

Max Activating Token Index: 997

Text #13

Max Range: 3.5911. Min Range: -3.5911

Max Act: 3.4657. Min Act: -0.1700

Data Index: 557595 (C4 (Web Text))

Max Activating Token Index: 44

Text #14

Max Range: 3.5911. Min Range: -3.5911

Max Act: 3.4187. Min Act: -0.1700

Data Index: 184565 (C4 (Web Text))

Max Activating Token Index: 765

Text #15

Max Range: 3.5911. Min Range: -3.5911

Max Act: 3.4344. Min Act: -0.1700

Data Index: 572771 (C4 (Web Text))

Max Activating Token Index: 357

Text #16

Max Range: 3.5911. Min Range: -3.5911

Max Act: 3.4344. Min Act: -0.1700

Data Index: 754832 (C4 (Web Text))

Max Activating Token Index: 513

Text #17

Max Range: 3.5911. Min Range: -3.5911

Max Act: 3.4344. Min Act: -0.1700

Data Index: 1111140 (C4 (Web Text))

Max Activating Token Index: 210

Text #18

Max Range: 3.5911. Min Range: -3.5911

Max Act: 3.4344. Min Act: -0.1700

Data Index: 147312 (C4 (Web Text))

Max Activating Token Index: 869

Text #19

Max Range: 3.5911. Min Range: -3.5911

Max Act: 3.4344. Min Act: -0.1700

Data Index: 760068 (C4 (Web Text))

Max Activating Token Index: 176