Model: GELU Model: 4 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 749 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('gelu-4l')

Text #0

Max Range: 5.7097. Min Range: -5.7097

Max Act: 5.7097. Min Act: -0.1700

Data Index: 1174267 (C4 (Web Text))

Max Activating Token Index: 149

Text #1

Max Range: 5.7097. Min Range: -5.7097

Max Act: 5.6785. Min Act: -0.1700

Data Index: 1293611 (C4 (Web Text))

Max Activating Token Index: 652

Text #2

Max Range: 5.7097. Min Range: -5.7097

Max Act: 5.5847. Min Act: -0.1700

Data Index: 35015 (C4 (Web Text))

Max Activating Token Index: 332

Text #3

Max Range: 5.7097. Min Range: -5.7097

Max Act: 5.5222. Min Act: -0.1700

Data Index: 479946 (C4 (Web Text))

Max Activating Token Index: 309

Text #4

Max Range: 5.7097. Min Range: -5.7097

Max Act: 5.5535. Min Act: -0.1700

Data Index: 1302763 (C4 (Web Text))

Max Activating Token Index: 551

Text #5

Max Range: 5.7097. Min Range: -5.7097

Max Act: 5.5535. Min Act: -0.1700

Data Index: 1060290 (C4 (Web Text))

Max Activating Token Index: 479

Text #6

Max Range: 5.7097. Min Range: -5.7097

Max Act: 5.5222. Min Act: -0.1700

Data Index: 1316378 (C4 (Web Text))

Max Activating Token Index: 306

Text #7

Max Range: 5.7097. Min Range: -5.7097

Max Act: 5.5222. Min Act: -0.1700

Data Index: 496647 (C4 (Web Text))

Max Activating Token Index: 612

Text #8

Max Range: 5.7097. Min Range: -5.7097

Max Act: 5.4910. Min Act: -0.1700

Data Index: 187953 (C4 (Web Text))

Max Activating Token Index: 174

Text #9

Max Range: 5.7097. Min Range: -5.7097

Max Act: 5.4597. Min Act: -0.1700

Data Index: 892766 (C4 (Web Text))

Max Activating Token Index: 945

Text #10

Max Range: 5.7097. Min Range: -5.7097

Max Act: 5.4597. Min Act: -0.1700

Data Index: 291860 (C4 (Web Text))

Max Activating Token Index: 342

Text #11

Max Range: 5.7097. Min Range: -5.7097

Max Act: 5.4285. Min Act: -0.1700

Data Index: 1319924 (C4 (Web Text))

Max Activating Token Index: 693

Text #12

Max Range: 5.7097. Min Range: -5.7097

Max Act: 5.4597. Min Act: -0.1700

Data Index: 692483 (C4 (Web Text))

Max Activating Token Index: 572

Text #13

Max Range: 5.7097. Min Range: -5.7097

Max Act: 5.4910. Min Act: -0.1700

Data Index: 1259006 (C4 (Web Text))

Max Activating Token Index: 737

Text #14

Max Range: 5.7097. Min Range: -5.7097

Max Act: 5.3972. Min Act: -0.1700

Data Index: 396952 (C4 (Web Text))

Max Activating Token Index: 843

Text #15

Max Range: 5.7097. Min Range: -5.7097

Max Act: 5.3972. Min Act: -0.1700

Data Index: 570469 (C4 (Web Text))

Max Activating Token Index: 324

Text #16

Max Range: 5.7097. Min Range: -5.7097

Max Act: 5.3972. Min Act: -0.1700

Data Index: 1013403 (C4 (Web Text))

Max Activating Token Index: 642

Text #17

Max Range: 5.7097. Min Range: -5.7097

Max Act: 5.3972. Min Act: -0.1700

Data Index: 787103 (C4 (Web Text))

Max Activating Token Index: 199

Text #18

Max Range: 5.7097. Min Range: -5.7097

Max Act: 5.2722. Min Act: -0.1700

Data Index: 1349441 (C4 (Web Text))

Max Activating Token Index: 666

Text #19

Max Range: 5.7097. Min Range: -5.7097

Max Act: 5.3347. Min Act: -0.1700

Data Index: 754264 (C4 (Web Text))

Max Activating Token Index: 611