Model: GELU Model: 3 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2032 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('gelu-3l')

Text #0

Max Range: 5.6598. Min Range: -5.6598

Max Act: 5.6598. Min Act: -0.1700

Data Index: 852332 (C4 (Web Text))

Max Activating Token Index: 101

Text #1

Max Range: 5.6598. Min Range: -5.6598

Max Act: 5.5660. Min Act: -0.1700

Data Index: 1017737 (C4 (Web Text))

Max Activating Token Index: 180

Text #2

Max Range: 5.6598. Min Range: -5.6598

Max Act: 5.4410. Min Act: -0.1700

Data Index: 1339244 (C4 (Web Text))

Max Activating Token Index: 155

Text #3

Max Range: 5.6598. Min Range: -5.6598

Max Act: 5.4098. Min Act: -0.1700

Data Index: 821094 (C4 (Web Text))

Max Activating Token Index: 571

Text #4

Max Range: 5.6598. Min Range: -5.6598

Max Act: 5.3473. Min Act: -0.1698

Data Index: 1069411 (C4 (Web Text))

Max Activating Token Index: 451

Text #5

Max Range: 5.6598. Min Range: -5.6598

Max Act: 5.3473. Min Act: -0.1700

Data Index: 668451 (C4 (Web Text))

Max Activating Token Index: 402

Text #6

Max Range: 5.6598. Min Range: -5.6598

Max Act: 5.3160. Min Act: -0.1700

Data Index: 140307 (C4 (Web Text))

Max Activating Token Index: 772

Text #7

Max Range: 5.6598. Min Range: -5.6598

Max Act: 5.3160. Min Act: -0.1700

Data Index: 1331891 (C4 (Web Text))

Max Activating Token Index: 701

Text #8

Max Range: 5.6598. Min Range: -5.6598

Max Act: 5.2535. Min Act: -0.1700

Data Index: 424947 (C4 (Web Text))

Max Activating Token Index: 189

Text #9

Max Range: 5.6598. Min Range: -5.6598

Max Act: 5.2223. Min Act: -0.1700

Data Index: 373694 (C4 (Web Text))

Max Activating Token Index: 584

Text #10

Max Range: 5.6598. Min Range: -5.6598

Max Act: 5.2535. Min Act: -0.1700

Data Index: 31214 (C4 (Web Text))

Max Activating Token Index: 981

Text #11

Max Range: 5.6598. Min Range: -5.6598

Max Act: 5.2535. Min Act: -0.1700

Data Index: 658192 (C4 (Web Text))

Max Activating Token Index: 580

Text #12

Max Range: 5.6598. Min Range: -5.6598

Max Act: 5.2223. Min Act: -0.1700

Data Index: 8454 (C4 (Web Text))

Max Activating Token Index: 990

Text #13

Max Range: 5.6598. Min Range: -5.6598

Max Act: 5.1910. Min Act: -0.1700

Data Index: 683913 (C4 (Web Text))

Max Activating Token Index: 741

Text #14

Max Range: 5.6598. Min Range: -5.6598

Max Act: 5.2535. Min Act: -0.1700

Data Index: 869088 (C4 (Web Text))

Max Activating Token Index: 185

Text #15

Max Range: 5.6598. Min Range: -5.6598

Max Act: 5.2223. Min Act: -0.1700

Data Index: 821188 (C4 (Web Text))

Max Activating Token Index: 226

Text #16

Max Range: 5.6598. Min Range: -5.6598

Max Act: 5.1598. Min Act: -0.1700

Data Index: 1055448 (C4 (Web Text))

Max Activating Token Index: 21

Text #17

Max Range: 5.6598. Min Range: -5.6598

Max Act: 5.1598. Min Act: -0.1700

Data Index: 386701 (C4 (Web Text))

Max Activating Token Index: 520

Text #18

Max Range: 5.6598. Min Range: -5.6598

Max Act: 5.1285. Min Act: -0.1700

Data Index: 3256 (C4 (Web Text))

Max Activating Token Index: 945

Text #19

Max Range: 5.6598. Min Range: -5.6598

Max Act: 5.1285. Min Act: -0.1700

Data Index: 644555 (C4 (Web Text))

Max Activating Token Index: 242