Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 481 in Layer 4

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.8001. Min Range: -0.8001

Max Act: 0.7919. Min Act: -0.0001

Data Index: 254710 (C4 (Web Text))

Max Activating Token Index: 863

Text #1

Max Range: 0.8001. Min Range: -0.8001

Max Act: 0.8001. Min Act: -0.0001

Data Index: 459464 (C4 (Web Text))

Max Activating Token Index: 1015

Text #2

Max Range: 0.8001. Min Range: -0.8001

Max Act: 0.7527. Min Act: -0.0001

Data Index: 1132606 (C4 (Web Text))

Max Activating Token Index: 228

Text #3

Max Range: 0.8001. Min Range: -0.8001

Max Act: 0.7183. Min Act: -0.0001

Data Index: 135098 (C4 (Web Text))

Max Activating Token Index: 678

Text #4

Max Range: 0.8001. Min Range: -0.8001

Max Act: 0.6781. Min Act: -0.0001

Data Index: 213473 (C4 (Web Text))

Max Activating Token Index: 253

Text #5

Max Range: 0.8001. Min Range: -0.8001

Max Act: 0.6585. Min Act: -0.0001

Data Index: 534058 (C4 (Web Text))

Max Activating Token Index: 780

Text #6

Max Range: 0.8001. Min Range: -0.8001

Max Act: 0.6569. Min Act: -0.0001

Data Index: 326969 (C4 (Web Text))

Max Activating Token Index: 500

Text #7

Max Range: 0.8001. Min Range: -0.8001

Max Act: 0.6134. Min Act: -0.0001

Data Index: 783417 (C4 (Web Text))

Max Activating Token Index: 698

Text #8

Max Range: 0.8001. Min Range: -0.8001

Max Act: 0.6610. Min Act: -0.0001

Data Index: 353059 (C4 (Web Text))

Max Activating Token Index: 848

Text #9

Max Range: 0.8001. Min Range: -0.8001

Max Act: 0.6171. Min Act: -0.0001

Data Index: 396765 (C4 (Web Text))

Max Activating Token Index: 565

Text #10

Max Range: 0.8001. Min Range: -0.8001

Max Act: 0.6501. Min Act: -0.0000

Data Index: 798093 (C4 (Web Text))

Max Activating Token Index: 31

Text #11

Max Range: 0.8001. Min Range: -0.8001

Max Act: 0.6082. Min Act: -0.0000

Data Index: 1336580 (C4 (Web Text))

Max Activating Token Index: 283

Text #12

Max Range: 0.8001. Min Range: -0.8001

Max Act: 0.5844. Min Act: -0.0001

Data Index: 237074 (C4 (Web Text))

Max Activating Token Index: 909

Text #13

Max Range: 0.8001. Min Range: -0.8001

Max Act: 0.5709. Min Act: -0.0001

Data Index: 1184081 (C4 (Web Text))

Max Activating Token Index: 188

Text #14

Max Range: 0.8001. Min Range: -0.8001

Max Act: 0.6740. Min Act: -0.0001

Data Index: 76465 (C4 (Web Text))

Max Activating Token Index: 152

Text #15

Max Range: 0.8001. Min Range: -0.8001

Max Act: 0.5958. Min Act: -0.0001

Data Index: 1113286 (C4 (Web Text))

Max Activating Token Index: 916

Text #16

Max Range: 0.8001. Min Range: -0.8001

Max Act: 0.6111. Min Act: -0.0000

Data Index: 334022 (C4 (Web Text))

Max Activating Token Index: 991

Text #17

Max Range: 0.8001. Min Range: -0.8001

Max Act: 0.6288. Min Act: -0.0001

Data Index: 94994 (C4 (Web Text))

Max Activating Token Index: 622

Text #18

Max Range: 0.8001. Min Range: -0.8001

Max Act: 0.5911. Min Act: -0.0001

Data Index: 798091 (C4 (Web Text))

Max Activating Token Index: 635

Text #19

Max Range: 0.8001. Min Range: -0.8001

Max Act: 0.6056. Min Act: -0.0000

Data Index: 208715 (C4 (Web Text))

Max Activating Token Index: 80