Model: GELU Model: 3 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1124 in Layer 1

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('gelu-3l')

Text #0

Max Range: 3.2258. Min Range: -3.2258

Max Act: 3.2258. Min Act: -0.1700

Data Index: 726454 (C4 (Web Text))

Max Activating Token Index: 556

Text #1

Max Range: 3.2258. Min Range: -3.2258

Max Act: 3.1786. Min Act: -0.1700

Data Index: 1342244 (C4 (Web Text))

Max Activating Token Index: 285

Text #2

Max Range: 3.2258. Min Range: -3.2258

Max Act: 3.1629. Min Act: -0.1700

Data Index: 570308 (C4 (Web Text))

Max Activating Token Index: 531

Text #3

Max Range: 3.2258. Min Range: -3.2258

Max Act: 3.1314. Min Act: -0.1700

Data Index: 1152671 (C4 (Web Text))

Max Activating Token Index: 743

Text #4

Max Range: 3.2258. Min Range: -3.2258

Max Act: 3.1471. Min Act: -0.1700

Data Index: 1131713 (C4 (Web Text))

Max Activating Token Index: 737

Text #5

Max Range: 3.2258. Min Range: -3.2258

Max Act: 3.1629. Min Act: -0.1700

Data Index: 901344 (C4 (Web Text))

Max Activating Token Index: 907

Text #6

Max Range: 3.2258. Min Range: -3.2258

Max Act: 3.1156. Min Act: -0.1700

Data Index: 368775 (C4 (Web Text))

Max Activating Token Index: 835

Text #7

Max Range: 3.2258. Min Range: -3.2258

Max Act: 3.1156. Min Act: -0.1700

Data Index: 38992 (C4 (Web Text))

Max Activating Token Index: 710

Text #8

Max Range: 3.2258. Min Range: -3.2258

Max Act: 3.0998. Min Act: -0.1700

Data Index: 368388 (C4 (Web Text))

Max Activating Token Index: 376

Text #9

Max Range: 3.2258. Min Range: -3.2258

Max Act: 3.0998. Min Act: -0.1700

Data Index: 339899 (C4 (Web Text))

Max Activating Token Index: 702

Text #10

Max Range: 3.2258. Min Range: -3.2258

Max Act: 3.1156. Min Act: -0.1700

Data Index: 769031 (C4 (Web Text))

Max Activating Token Index: 539

Text #11

Max Range: 3.2258. Min Range: -3.2258

Max Act: 3.0841. Min Act: -0.1700

Data Index: 1219152 (C4 (Web Text))

Max Activating Token Index: 534

Text #12

Max Range: 3.2258. Min Range: -3.2258

Max Act: 3.0683. Min Act: -0.1700

Data Index: 1135026 (C4 (Web Text))

Max Activating Token Index: 171

Text #13

Max Range: 3.2258. Min Range: -3.2258

Max Act: 3.0841. Min Act: -0.1700

Data Index: 896312 (C4 (Web Text))

Max Activating Token Index: 783

Text #14

Max Range: 3.2258. Min Range: -3.2258

Max Act: 3.0841. Min Act: -0.1700

Data Index: 2669 (C4 (Web Text))

Max Activating Token Index: 902

Text #15

Max Range: 3.2258. Min Range: -3.2258

Max Act: 3.0841. Min Act: -0.1700

Data Index: 988 (C4 (Web Text))

Max Activating Token Index: 959

Text #16

Max Range: 3.2258. Min Range: -3.2258

Max Act: 3.0841. Min Act: -0.1700

Data Index: 143322 (C4 (Web Text))

Max Activating Token Index: 653

Text #17

Max Range: 3.2258. Min Range: -3.2258

Max Act: 3.0683. Min Act: -0.1700

Data Index: 594264 (C4 (Web Text))

Max Activating Token Index: 909

Text #18

Max Range: 3.2258. Min Range: -3.2258

Max Act: 3.0841. Min Act: -0.1700

Data Index: 750737 (C4 (Web Text))

Max Activating Token Index: 707

Text #19

Max Range: 3.2258. Min Range: -3.2258

Max Act: 3.0683. Min Act: -0.1700

Data Index: 1124397 (C4 (Web Text))

Max Activating Token Index: 725