Model: GELU Model: 3 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 514 in Layer 1

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('gelu-3l')

Text #0

Max Range: 3.1838. Min Range: -3.1838

Max Act: 3.1838. Min Act: -0.1700

Data Index: 1269206 (C4 (Web Text))

Max Activating Token Index: 731

Text #1

Max Range: 3.1838. Min Range: -3.1838

Max Act: 3.1680. Min Act: -0.1700

Data Index: 18660 (C4 (Web Text))

Max Activating Token Index: 262

Text #2

Max Range: 3.1838. Min Range: -3.1838

Max Act: 3.1523. Min Act: -0.1700

Data Index: 899713 (C4 (Web Text))

Max Activating Token Index: 159

Text #3

Max Range: 3.1838. Min Range: -3.1838

Max Act: 3.1208. Min Act: -0.1700

Data Index: 53498 (C4 (Web Text))

Max Activating Token Index: 571

Text #4

Max Range: 3.1838. Min Range: -3.1838

Max Act: 3.1208. Min Act: -0.1700

Data Index: 958449 (C4 (Web Text))

Max Activating Token Index: 605

Text #5

Max Range: 3.1838. Min Range: -3.1838

Max Act: 3.1208. Min Act: -0.1700

Data Index: 93506 (C4 (Web Text))

Max Activating Token Index: 995

Text #6

Max Range: 3.1838. Min Range: -3.1838

Max Act: 3.1366. Min Act: -0.1700

Data Index: 987020 (C4 (Web Text))

Max Activating Token Index: 183

Text #7

Max Range: 3.1838. Min Range: -3.1838

Max Act: 3.1208. Min Act: -0.1700

Data Index: 594462 (C4 (Web Text))

Max Activating Token Index: 491

Text #8

Max Range: 3.1838. Min Range: -3.1838

Max Act: 3.1050. Min Act: -0.1700

Data Index: 521454 (C4 (Web Text))

Max Activating Token Index: 609

Text #9

Max Range: 3.1838. Min Range: -3.1838

Max Act: 3.1050. Min Act: -0.1700

Data Index: 531596 (C4 (Web Text))

Max Activating Token Index: 225

Text #10

Max Range: 3.1838. Min Range: -3.1838

Max Act: 3.0893. Min Act: -0.1700

Data Index: 664256 (C4 (Web Text))

Max Activating Token Index: 658

Text #11

Max Range: 3.1838. Min Range: -3.1838

Max Act: 3.0893. Min Act: -0.1700

Data Index: 848951 (C4 (Web Text))

Max Activating Token Index: 369

Text #12

Max Range: 3.1838. Min Range: -3.1838

Max Act: 3.0735. Min Act: -0.1700

Data Index: 1326167 (C4 (Web Text))

Max Activating Token Index: 313

Text #13

Max Range: 3.1838. Min Range: -3.1838

Max Act: 3.0577. Min Act: -0.1700

Data Index: 487690 (C4 (Web Text))

Max Activating Token Index: 437

Text #14

Max Range: 3.1838. Min Range: -3.1838

Max Act: 3.0577. Min Act: -0.1700

Data Index: 1219740 (C4 (Web Text))

Max Activating Token Index: 513

Text #15

Max Range: 3.1838. Min Range: -3.1838

Max Act: 3.0893. Min Act: -0.1700

Data Index: 1358025 (C4 (Web Text))

Max Activating Token Index: 413

Text #16

Max Range: 3.1838. Min Range: -3.1838

Max Act: 3.0735. Min Act: -0.1700

Data Index: 656834 (C4 (Web Text))

Max Activating Token Index: 656

Text #17

Max Range: 3.1838. Min Range: -3.1838

Max Act: 3.0735. Min Act: -0.1700

Data Index: 14286 (C4 (Web Text))

Max Activating Token Index: 676

Text #18

Max Range: 3.1838. Min Range: -3.1838

Max Act: 3.0577. Min Act: -0.1700

Data Index: 193298 (C4 (Web Text))

Max Activating Token Index: 330

Text #19

Max Range: 3.1838. Min Range: -3.1838

Max Act: 3.0419. Min Act: -0.1700

Data Index: 905135 (C4 (Web Text))

Max Activating Token Index: 404