Model: GELU Model: 1 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 960 in Layer 0

Load this data into an Interactive Neuroscope

See Documentation here

Hooked Transformer Loading: HookedTransformer.from_pretrained('gelu-1l')

Click the toggle to see the full text!

Text #0

Max Range: 3.3281. Min Range: -3.3281

Max Act: 3.3281. Min Act: -0.1700

Data Index: 1400125 (Python Code)

Max Activating Token Index: 189

Text #1

Max Range: 3.3281. Min Range: -3.3281

Max Act: 3.2675. Min Act: -0.1700

Data Index: 1514765 (Python Code)

Max Activating Token Index: 189

Text #2

Max Range: 3.3281. Min Range: -3.3281

Max Act: 3.1982. Min Act: -0.1700

Data Index: 693005 (C4 (Web Text))

Max Activating Token Index: 459

Text #3

Max Range: 3.3281. Min Range: -3.3281

Max Act: 3.1903. Min Act: -0.1697

Data Index: 1534421 (Python Code)

Max Activating Token Index: 683

Text #4

Max Range: 3.3281. Min Range: -3.3281

Max Act: 3.1629. Min Act: -0.1700

Data Index: 1521971 (Python Code)

Max Activating Token Index: 574

Text #5

Max Range: 3.3281. Min Range: -3.3281

Max Act: 3.1609. Min Act: -0.1700

Data Index: 1656599 (Python Code)

Max Activating Token Index: 243

Text #6

Max Range: 3.3281. Min Range: -3.3281

Max Act: 3.1273. Min Act: -0.1566

Data Index: 1574681 (Python Code)

Max Activating Token Index: 495

Text #7

Max Range: 3.3281. Min Range: -3.3281

Max Act: 3.1210. Min Act: -0.1695

Data Index: 1527339 (Python Code)

Max Activating Token Index: 665

Text #8

Max Range: 3.3281. Min Range: -3.3281

Max Act: 3.1020. Min Act: -0.1641

Data Index: 1543111 (Python Code)

Max Activating Token Index: 564

Text #9

Max Range: 3.3281. Min Range: -3.3281

Max Act: 3.1074. Min Act: -0.1699

Data Index: 1396271 (Python Code)

Max Activating Token Index: 851

Text #10

Max Range: 3.3281. Min Range: -3.3281

Max Act: 3.1042. Min Act: -0.1071

Data Index: 1440087 (Python Code)

Max Activating Token Index: 563

Text #11

Max Range: 3.3281. Min Range: -3.3281

Max Act: 3.1137. Min Act: -0.1699

Data Index: 1593801 (Python Code)

Max Activating Token Index: 462

Text #12

Max Range: 3.3281. Min Range: -3.3281

Max Act: 3.1005. Min Act: -0.1481

Data Index: 1410884 (Python Code)

Max Activating Token Index: 110

Text #13

Max Range: 3.3281. Min Range: -3.3281

Max Act: 3.0962. Min Act: -0.1686

Data Index: 1468088 (Python Code)

Max Activating Token Index: 186

Text #14

Max Range: 3.3281. Min Range: -3.3281

Max Act: 3.0870. Min Act: -0.1700

Data Index: 1375346 (Python Code)

Max Activating Token Index: 810

Text #15

Max Range: 3.3281. Min Range: -3.3281

Max Act: 3.0717. Min Act: -0.1072

Data Index: 1519956 (Python Code)

Max Activating Token Index: 83

Text #16

Max Range: 3.3281. Min Range: -3.3281

Max Act: 3.0772. Min Act: -0.1700

Data Index: 1544723 (Python Code)

Max Activating Token Index: 819

Text #17

Max Range: 3.3281. Min Range: -3.3281

Max Act: 3.0791. Min Act: -0.1700

Data Index: 1550157 (Python Code)

Max Activating Token Index: 767

Text #18

Max Range: 3.3281. Min Range: -3.3281

Max Act: 3.0545. Min Act: -0.1699

Data Index: 1582722 (Python Code)

Max Activating Token Index: 65

Text #19

Max Range: 3.3281. Min Range: -3.3281

Max Act: 3.0522. Min Act: -0.1697

Data Index: 1593898 (Python Code)

Max Activating Token Index: 286