Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4110 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.0849. Min Range: -1.0849

Max Act: 1.0849. Min Act: -0.0002

Data Index: 413269 (C4 (Web Text))

Max Activating Token Index: 951

Text #1

Max Range: 1.0849. Min Range: -1.0849

Max Act: 1.0066. Min Act: -0.0002

Data Index: 1325459 (C4 (Web Text))

Max Activating Token Index: 85

Text #2

Max Range: 1.0849. Min Range: -1.0849

Max Act: 1.0498. Min Act: -0.0002

Data Index: 384832 (C4 (Web Text))

Max Activating Token Index: 799

Text #3

Max Range: 1.0849. Min Range: -1.0849

Max Act: 1.0760. Min Act: -0.0002

Data Index: 1041428 (C4 (Web Text))

Max Activating Token Index: 91

Text #4

Max Range: 1.0849. Min Range: -1.0849

Max Act: 1.0691. Min Act: -0.0002

Data Index: 807323 (C4 (Web Text))

Max Activating Token Index: 624

Text #5

Max Range: 1.0849. Min Range: -1.0849

Max Act: 1.0268. Min Act: -0.0002

Data Index: 1071614 (C4 (Web Text))

Max Activating Token Index: 528

Text #6

Max Range: 1.0849. Min Range: -1.0849

Max Act: 1.0023. Min Act: -0.0002

Data Index: 454360 (C4 (Web Text))

Max Activating Token Index: 175

Text #7

Max Range: 1.0849. Min Range: -1.0849

Max Act: 0.9585. Min Act: -0.0002

Data Index: 1005533 (C4 (Web Text))

Max Activating Token Index: 19

Text #8

Max Range: 1.0849. Min Range: -1.0849

Max Act: 0.9985. Min Act: -0.0002

Data Index: 1126982 (C4 (Web Text))

Max Activating Token Index: 884

Text #9

Max Range: 1.0849. Min Range: -1.0849

Max Act: 0.9568. Min Act: -0.0002

Data Index: 253981 (C4 (Web Text))

Max Activating Token Index: 540

Text #10

Max Range: 1.0849. Min Range: -1.0849

Max Act: 0.8775. Min Act: -0.0002

Data Index: 663532 (C4 (Web Text))

Max Activating Token Index: 659

Text #11

Max Range: 1.0849. Min Range: -1.0849

Max Act: 0.9913. Min Act: -0.0002

Data Index: 1358570 (C4 (Web Text))

Max Activating Token Index: 267

Text #12

Max Range: 1.0849. Min Range: -1.0849

Max Act: 0.9750. Min Act: -0.0002

Data Index: 1111808 (C4 (Web Text))

Max Activating Token Index: 433

Text #13

Max Range: 1.0849. Min Range: -1.0849

Max Act: 0.9420. Min Act: -0.0002

Data Index: 781533 (C4 (Web Text))

Max Activating Token Index: 256

Text #14

Max Range: 1.0849. Min Range: -1.0849

Max Act: 0.9973. Min Act: -0.0002

Data Index: 1285850 (C4 (Web Text))

Max Activating Token Index: 909

Text #15

Max Range: 1.0849. Min Range: -1.0849

Max Act: 0.9671. Min Act: -0.0002

Data Index: 212850 (C4 (Web Text))

Max Activating Token Index: 27

Text #16

Max Range: 1.0849. Min Range: -1.0849

Max Act: 0.8725. Min Act: -0.0002

Data Index: 267251 (C4 (Web Text))

Max Activating Token Index: 886

Text #17

Max Range: 1.0849. Min Range: -1.0849

Max Act: 0.9037. Min Act: -0.0002

Data Index: 442326 (C4 (Web Text))

Max Activating Token Index: 157

Text #18

Max Range: 1.0849. Min Range: -1.0849

Max Act: 1.0128. Min Act: -0.0002

Data Index: 1288576 (C4 (Web Text))

Max Activating Token Index: 19

Text #19

Max Range: 1.0849. Min Range: -1.0849

Max Act: 0.9388. Min Act: -0.0002

Data Index: 262636 (C4 (Web Text))

Max Activating Token Index: 948