Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 432 in Layer 11

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.0111. Min Range: -1.0111

Max Act: 1.0111. Min Act: -0.0000

Data Index: 451407 (C4 (Web Text))

Max Activating Token Index: 116

Text #1

Max Range: 1.0111. Min Range: -1.0111

Max Act: 0.8498. Min Act: -0.0000

Data Index: 905055 (C4 (Web Text))

Max Activating Token Index: 1011

Text #2

Max Range: 1.0111. Min Range: -1.0111

Max Act: 0.7652. Min Act: -0.0000

Data Index: 1144584 (C4 (Web Text))

Max Activating Token Index: 315

Text #3

Max Range: 1.0111. Min Range: -1.0111

Max Act: 0.7543. Min Act: -0.0000

Data Index: 1174414 (C4 (Web Text))

Max Activating Token Index: 540

Text #4

Max Range: 1.0111. Min Range: -1.0111

Max Act: 0.7751. Min Act: -0.0000

Data Index: 37467 (C4 (Web Text))

Max Activating Token Index: 91

Text #5

Max Range: 1.0111. Min Range: -1.0111

Max Act: 0.6814. Min Act: -0.0000

Data Index: 436834 (C4 (Web Text))

Max Activating Token Index: 954

Text #6

Max Range: 1.0111. Min Range: -1.0111

Max Act: 0.7105. Min Act: -0.0000

Data Index: 1333674 (C4 (Web Text))

Max Activating Token Index: 143

Text #7

Max Range: 1.0111. Min Range: -1.0111

Max Act: 0.6850. Min Act: -0.0000

Data Index: 567164 (C4 (Web Text))

Max Activating Token Index: 991

Text #8

Max Range: 1.0111. Min Range: -1.0111

Max Act: 0.6739. Min Act: -0.0000

Data Index: 1282389 (C4 (Web Text))

Max Activating Token Index: 423

Text #9

Max Range: 1.0111. Min Range: -1.0111

Max Act: 0.6594. Min Act: -0.0000

Data Index: 504819 (C4 (Web Text))

Max Activating Token Index: 903

Text #10

Max Range: 1.0111. Min Range: -1.0111

Max Act: 0.6654. Min Act: -0.0000

Data Index: 927805 (C4 (Web Text))

Max Activating Token Index: 715

Text #11

Max Range: 1.0111. Min Range: -1.0111

Max Act: 0.6304. Min Act: -0.0000

Data Index: 241718 (C4 (Web Text))

Max Activating Token Index: 338

Text #12

Max Range: 1.0111. Min Range: -1.0111

Max Act: 0.5983. Min Act: -0.0000

Data Index: 842890 (C4 (Web Text))

Max Activating Token Index: 326

Text #13

Max Range: 1.0111. Min Range: -1.0111

Max Act: 0.5258. Min Act: -0.0000

Data Index: 541161 (C4 (Web Text))

Max Activating Token Index: 366

Text #14

Max Range: 1.0111. Min Range: -1.0111

Max Act: 0.5582. Min Act: -0.0000

Data Index: 301884 (C4 (Web Text))

Max Activating Token Index: 947

Text #15

Max Range: 1.0111. Min Range: -1.0111

Max Act: 0.5291. Min Act: -0.0000

Data Index: 1262557 (C4 (Web Text))

Max Activating Token Index: 885

Text #16

Max Range: 1.0111. Min Range: -1.0111

Max Act: 0.5477. Min Act: -0.0000

Data Index: 119285 (C4 (Web Text))

Max Activating Token Index: 501

Text #17

Max Range: 1.0111. Min Range: -1.0111

Max Act: 0.5378. Min Act: -0.0000

Data Index: 1080026 (C4 (Web Text))

Max Activating Token Index: 166

Text #18

Max Range: 1.0111. Min Range: -1.0111

Max Act: 0.4926. Min Act: -0.0000

Data Index: 567085 (C4 (Web Text))

Max Activating Token Index: 610

Text #19

Max Range: 1.0111. Min Range: -1.0111

Max Act: 0.5095. Min Act: -0.0000

Data Index: 1064354 (C4 (Web Text))

Max Activating Token Index: 449