Model: GELU Model: 4 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1852 in Layer 3

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('gelu-4l')

Text #0

Max Range: 10.0705. Min Range: -10.0705

Max Act: 10.0705. Min Act: -0.0979

Data Index: 748201 (C4 (Web Text))

Max Activating Token Index: 673

Text #1

Max Range: 10.0705. Min Range: -10.0705

Max Act: 10.0080. Min Act: -0.1694

Data Index: 689540 (C4 (Web Text))

Max Activating Token Index: 610

Text #2

Max Range: 10.0705. Min Range: -10.0705

Max Act: 9.5705. Min Act: -0.1621

Data Index: 396012 (C4 (Web Text))

Max Activating Token Index: 945

Text #3

Max Range: 10.0705. Min Range: -10.0705

Max Act: 9.3830. Min Act: -0.0954

Data Index: 1220715 (C4 (Web Text))

Max Activating Token Index: 471

Text #4

Max Range: 10.0705. Min Range: -10.0705

Max Act: 9.4455. Min Act: -0.0713

Data Index: 43218 (C4 (Web Text))

Max Activating Token Index: 331

Text #5

Max Range: 10.0705. Min Range: -10.0705

Max Act: 9.5705. Min Act: -0.1606

Data Index: 1075335 (C4 (Web Text))

Max Activating Token Index: 510

Text #6

Max Range: 10.0705. Min Range: -10.0705

Max Act: 9.4455. Min Act: -0.1606

Data Index: 565126 (C4 (Web Text))

Max Activating Token Index: 833

Text #7

Max Range: 10.0705. Min Range: -10.0705

Max Act: 9.3830. Min Act: -0.1666

Data Index: 574453 (C4 (Web Text))

Max Activating Token Index: 256

Text #8

Max Range: 10.0705. Min Range: -10.0705

Max Act: 9.4455. Min Act: -0.0979

Data Index: 340794 (C4 (Web Text))

Max Activating Token Index: 727

Text #9

Max Range: 10.0705. Min Range: -10.0705

Max Act: 9.5080. Min Act: -0.1094

Data Index: 1298522 (C4 (Web Text))

Max Activating Token Index: 567

Text #10

Max Range: 10.0705. Min Range: -10.0705

Max Act: 9.4455. Min Act: -0.1696

Data Index: 153008 (C4 (Web Text))

Max Activating Token Index: 227

Text #11

Max Range: 10.0705. Min Range: -10.0705

Max Act: 9.2580. Min Act: -0.1606

Data Index: 977850 (C4 (Web Text))

Max Activating Token Index: 219

Text #12

Max Range: 10.0705. Min Range: -10.0705

Max Act: 9.3205. Min Act: -0.1447

Data Index: 1306866 (C4 (Web Text))

Max Activating Token Index: 852

Text #13

Max Range: 10.0705. Min Range: -10.0705

Max Act: 9.3205. Min Act: -0.1606

Data Index: 157928 (C4 (Web Text))

Max Activating Token Index: 383

Text #14

Max Range: 10.0705. Min Range: -10.0705

Max Act: 9.3830. Min Act: -0.1598

Data Index: 484760 (C4 (Web Text))

Max Activating Token Index: 533

Text #15

Max Range: 10.0705. Min Range: -10.0705

Max Act: 9.3830. Min Act: -0.0713

Data Index: 768319 (C4 (Web Text))

Max Activating Token Index: 217

Text #16

Max Range: 10.0705. Min Range: -10.0705

Max Act: 9.3205. Min Act: -0.1072

Data Index: 317865 (C4 (Web Text))

Max Activating Token Index: 88

Text #17

Max Range: 10.0705. Min Range: -10.0705

Max Act: 9.2580. Min Act: -0.1486

Data Index: 538766 (C4 (Web Text))

Max Activating Token Index: 673

Text #18

Max Range: 10.0705. Min Range: -10.0705

Max Act: 9.1955. Min Act: -0.1274

Data Index: 848112 (C4 (Web Text))

Max Activating Token Index: 309

Text #19

Max Range: 10.0705. Min Range: -10.0705

Max Act: 9.2580. Min Act: -0.1699

Data Index: 346591 (C4 (Web Text))

Max Activating Token Index: 559