Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1268 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.2752. Min Range: -0.2752

Max Act: 0.2752. Min Act: -0.0000

Data Index: 359110 (C4 (Web Text))

Max Activating Token Index: 985

Text #1

Max Range: 0.2752. Min Range: -0.2752

Max Act: 0.2638. Min Act: -0.0000

Data Index: 928358 (C4 (Web Text))

Max Activating Token Index: 119

Text #2

Max Range: 0.2752. Min Range: -0.2752

Max Act: 0.2521. Min Act: -0.0000

Data Index: 756092 (C4 (Web Text))

Max Activating Token Index: 103

Text #3

Max Range: 0.2752. Min Range: -0.2752

Max Act: 0.2368. Min Act: -0.0000

Data Index: 1223447 (C4 (Web Text))

Max Activating Token Index: 742

Text #4

Max Range: 0.2752. Min Range: -0.2752

Max Act: 0.2588. Min Act: -0.0000

Data Index: 800352 (C4 (Web Text))

Max Activating Token Index: 256

Text #5

Max Range: 0.2752. Min Range: -0.2752

Max Act: 0.2131. Min Act: -0.0000

Data Index: 222493 (C4 (Web Text))

Max Activating Token Index: 1008

Text #6

Max Range: 0.2752. Min Range: -0.2752

Max Act: 0.2042. Min Act: -0.0000

Data Index: 1190466 (C4 (Web Text))

Max Activating Token Index: 149

Text #7

Max Range: 0.2752. Min Range: -0.2752

Max Act: 0.2030. Min Act: -0.0000

Data Index: 166620 (C4 (Web Text))

Max Activating Token Index: 263

Text #8

Max Range: 0.2752. Min Range: -0.2752

Max Act: 0.2203. Min Act: -0.0000

Data Index: 1159631 (C4 (Web Text))

Max Activating Token Index: 852

Text #9

Max Range: 0.2752. Min Range: -0.2752

Max Act: 0.1982. Min Act: -0.0000

Data Index: 749790 (C4 (Web Text))

Max Activating Token Index: 992

Text #10

Max Range: 0.2752. Min Range: -0.2752

Max Act: 0.1991. Min Act: -0.0000

Data Index: 666862 (C4 (Web Text))

Max Activating Token Index: 197

Text #11

Max Range: 0.2752. Min Range: -0.2752

Max Act: 0.1969. Min Act: -0.0000

Data Index: 675354 (C4 (Web Text))

Max Activating Token Index: 354

Text #12

Max Range: 0.2752. Min Range: -0.2752

Max Act: 0.1951. Min Act: -0.0000

Data Index: 1248592 (C4 (Web Text))

Max Activating Token Index: 372

Text #13

Max Range: 0.2752. Min Range: -0.2752

Max Act: 0.1927. Min Act: -0.0000

Data Index: 193183 (C4 (Web Text))

Max Activating Token Index: 16

Text #14

Max Range: 0.2752. Min Range: -0.2752

Max Act: 0.1959. Min Act: -0.0000

Data Index: 875896 (C4 (Web Text))

Max Activating Token Index: 762

Text #15

Max Range: 0.2752. Min Range: -0.2752

Max Act: 0.1882. Min Act: -0.0000

Data Index: 425978 (C4 (Web Text))

Max Activating Token Index: 318

Text #16

Max Range: 0.2752. Min Range: -0.2752

Max Act: 0.1884. Min Act: -0.0000

Data Index: 1334771 (C4 (Web Text))

Max Activating Token Index: 64

Text #17

Max Range: 0.2752. Min Range: -0.2752

Max Act: 0.1892. Min Act: -0.0000

Data Index: 254363 (C4 (Web Text))

Max Activating Token Index: 718

Text #18

Max Range: 0.2752. Min Range: -0.2752

Max Act: 0.1814. Min Act: -0.0000

Data Index: 1355725 (C4 (Web Text))

Max Activating Token Index: 727

Text #19

Max Range: 0.2752. Min Range: -0.2752

Max Act: 0.1869. Min Act: -0.0000

Data Index: 229110 (C4 (Web Text))

Max Activating Token Index: 361