Model: GELU Model: 2 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1218 in Layer 1

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('gelu-2l')

Text #0

Max Range: 3.9967. Min Range: -3.9967

Max Act: 3.9967. Min Act: -0.1700

Data Index: 737692 (C4 (Web Text))

Max Activating Token Index: 469

Text #1

Max Range: 3.9967. Min Range: -3.9967

Max Act: 3.9341. Min Act: -0.1700

Data Index: 1040597 (C4 (Web Text))

Max Activating Token Index: 304

Text #2

Max Range: 3.9967. Min Range: -3.9967

Max Act: 3.8716. Min Act: -0.1700

Data Index: 946398 (C4 (Web Text))

Max Activating Token Index: 732

Text #3

Max Range: 3.9967. Min Range: -3.9967

Max Act: 3.8559. Min Act: -0.1700

Data Index: 204159 (C4 (Web Text))

Max Activating Token Index: 716

Text #4

Max Range: 3.9967. Min Range: -3.9967

Max Act: 3.8247. Min Act: -0.1700

Data Index: 692765 (C4 (Web Text))

Max Activating Token Index: 551

Text #5

Max Range: 3.9967. Min Range: -3.9967

Max Act: 3.7777. Min Act: -0.1700

Data Index: 1084892 (C4 (Web Text))

Max Activating Token Index: 761

Text #6

Max Range: 3.9967. Min Range: -3.9967

Max Act: 3.8403. Min Act: -0.1700

Data Index: 1082028 (C4 (Web Text))

Max Activating Token Index: 351

Text #7

Max Range: 3.9967. Min Range: -3.9967

Max Act: 3.7777. Min Act: -0.1700

Data Index: 670917 (C4 (Web Text))

Max Activating Token Index: 544

Text #8

Max Range: 3.9967. Min Range: -3.9967

Max Act: 3.7308. Min Act: -0.1700

Data Index: 451956 (C4 (Web Text))

Max Activating Token Index: 60

Text #9

Max Range: 3.9967. Min Range: -3.9967

Max Act: 3.6839. Min Act: -0.1700

Data Index: 846533 (C4 (Web Text))

Max Activating Token Index: 485

Text #10

Max Range: 3.9967. Min Range: -3.9967

Max Act: 3.6995. Min Act: -0.1700

Data Index: 635323 (C4 (Web Text))

Max Activating Token Index: 635

Text #11

Max Range: 3.9967. Min Range: -3.9967

Max Act: 3.7152. Min Act: -0.1700

Data Index: 110729 (C4 (Web Text))

Max Activating Token Index: 560

Text #12

Max Range: 3.9967. Min Range: -3.9967

Max Act: 3.6682. Min Act: -0.1700

Data Index: 678012 (C4 (Web Text))

Max Activating Token Index: 947

Text #13

Max Range: 3.9967. Min Range: -3.9967

Max Act: 3.6526. Min Act: -0.1700

Data Index: 24167 (C4 (Web Text))

Max Activating Token Index: 494

Text #14

Max Range: 3.9967. Min Range: -3.9967

Max Act: 3.6839. Min Act: -0.1700

Data Index: 852544 (C4 (Web Text))

Max Activating Token Index: 237

Text #15

Max Range: 3.9967. Min Range: -3.9967

Max Act: 3.6213. Min Act: -0.1700

Data Index: 722786 (C4 (Web Text))

Max Activating Token Index: 271

Text #16

Max Range: 3.9967. Min Range: -3.9967

Max Act: 3.6369. Min Act: -0.1700

Data Index: 1192534 (C4 (Web Text))

Max Activating Token Index: 132

Text #17

Max Range: 3.9967. Min Range: -3.9967

Max Act: 3.6213. Min Act: -0.1700

Data Index: 111466 (C4 (Web Text))

Max Activating Token Index: 295

Text #18

Max Range: 3.9967. Min Range: -3.9967

Max Act: 3.6369. Min Act: -0.1700

Data Index: 810975 (C4 (Web Text))

Max Activating Token Index: 622

Text #19

Max Range: 3.9967. Min Range: -3.9967

Max Act: 3.6213. Min Act: -0.1700

Data Index: 901486 (C4 (Web Text))

Max Activating Token Index: 491