Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2537 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.1013. Min Range: -1.1013

Max Act: 1.0862. Min Act: -0.0000

Data Index: 596780 (C4 (Web Text))

Max Activating Token Index: 111

Text #1

Max Range: 1.1013. Min Range: -1.1013

Max Act: 1.0782. Min Act: -0.0000

Data Index: 520408 (C4 (Web Text))

Max Activating Token Index: 458

Text #2

Max Range: 1.1013. Min Range: -1.1013

Max Act: 1.0844. Min Act: -0.0000

Data Index: 1276085 (C4 (Web Text))

Max Activating Token Index: 190

Text #3

Max Range: 1.1013. Min Range: -1.1013

Max Act: 1.1013. Min Act: -0.0000

Data Index: 818625 (C4 (Web Text))

Max Activating Token Index: 813

Text #4

Max Range: 1.1013. Min Range: -1.1013

Max Act: 1.0649. Min Act: -0.0000

Data Index: 134011 (C4 (Web Text))

Max Activating Token Index: 134

Text #5

Max Range: 1.1013. Min Range: -1.1013

Max Act: 1.0516. Min Act: -0.0000

Data Index: 394132 (C4 (Web Text))

Max Activating Token Index: 691

Text #6

Max Range: 1.1013. Min Range: -1.1013

Max Act: 1.0493. Min Act: -0.0000

Data Index: 402018 (C4 (Web Text))

Max Activating Token Index: 879

Text #7

Max Range: 1.1013. Min Range: -1.1013

Max Act: 1.0321. Min Act: -0.0000

Data Index: 907705 (C4 (Web Text))

Max Activating Token Index: 238

Text #8

Max Range: 1.1013. Min Range: -1.1013

Max Act: 0.9945. Min Act: -0.0000

Data Index: 904491 (C4 (Web Text))

Max Activating Token Index: 400

Text #9

Max Range: 1.1013. Min Range: -1.1013

Max Act: 0.9689. Min Act: -0.0000

Data Index: 1236669 (C4 (Web Text))

Max Activating Token Index: 902

Text #10

Max Range: 1.1013. Min Range: -1.1013

Max Act: 0.9580. Min Act: -0.0000

Data Index: 1145709 (C4 (Web Text))

Max Activating Token Index: 221

Text #11

Max Range: 1.1013. Min Range: -1.1013

Max Act: 0.9831. Min Act: -0.0000

Data Index: 391929 (C4 (Web Text))

Max Activating Token Index: 340

Text #12

Max Range: 1.1013. Min Range: -1.1013

Max Act: 0.9825. Min Act: -0.0000

Data Index: 169688 (C4 (Web Text))

Max Activating Token Index: 924

Text #13

Max Range: 1.1013. Min Range: -1.1013

Max Act: 1.0058. Min Act: -0.0000

Data Index: 514072 (C4 (Web Text))

Max Activating Token Index: 407

Text #14

Max Range: 1.1013. Min Range: -1.1013

Max Act: 1.0341. Min Act: -0.0000

Data Index: 1265542 (C4 (Web Text))

Max Activating Token Index: 190

Text #15

Max Range: 1.1013. Min Range: -1.1013

Max Act: 0.9664. Min Act: -0.0000

Data Index: 353977 (C4 (Web Text))

Max Activating Token Index: 453

Text #16

Max Range: 1.1013. Min Range: -1.1013

Max Act: 0.9833. Min Act: -0.0000

Data Index: 1301960 (C4 (Web Text))

Max Activating Token Index: 409

Text #17

Max Range: 1.1013. Min Range: -1.1013

Max Act: 0.9802. Min Act: -0.0000

Data Index: 1131827 (C4 (Web Text))

Max Activating Token Index: 584

Text #18

Max Range: 1.1013. Min Range: -1.1013

Max Act: 0.9643. Min Act: -0.0000

Data Index: 1034775 (C4 (Web Text))

Max Activating Token Index: 886

Text #19

Max Range: 1.1013. Min Range: -1.1013

Max Act: 0.9900. Min Act: -0.0000

Data Index: 461298 (C4 (Web Text))

Max Activating Token Index: 296