Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4424 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.1113. Min Range: -0.1113

Max Act: 0.1113. Min Act: -0.0002

Data Index: 481371 (C4 (Web Text))

Max Activating Token Index: 71

Text #1

Max Range: 0.1113. Min Range: -0.1113

Max Act: 0.1006. Min Act: -0.0002

Data Index: 730742 (C4 (Web Text))

Max Activating Token Index: 209

Text #2

Max Range: 0.1113. Min Range: -0.1113

Max Act: 0.1092. Min Act: -0.0002

Data Index: 878235 (C4 (Web Text))

Max Activating Token Index: 628

Text #3

Max Range: 0.1113. Min Range: -0.1113

Max Act: 0.1033. Min Act: -0.0002

Data Index: 1328221 (C4 (Web Text))

Max Activating Token Index: 78

Text #4

Max Range: 0.1113. Min Range: -0.1113

Max Act: 0.1027. Min Act: -0.0002

Data Index: 526083 (C4 (Web Text))

Max Activating Token Index: 524

Text #5

Max Range: 0.1113. Min Range: -0.1113

Max Act: 0.1054. Min Act: -0.0002

Data Index: 106016 (C4 (Web Text))

Max Activating Token Index: 31

Text #6

Max Range: 0.1113. Min Range: -0.1113

Max Act: 0.1030. Min Act: -0.0002

Data Index: 451077 (C4 (Web Text))

Max Activating Token Index: 319

Text #7

Max Range: 0.1113. Min Range: -0.1113

Max Act: 0.0832. Min Act: -0.0002

Data Index: 378598 (C4 (Web Text))

Max Activating Token Index: 118

Text #8

Max Range: 0.1113. Min Range: -0.1113

Max Act: 0.0954. Min Act: -0.0002

Data Index: 817072 (C4 (Web Text))

Max Activating Token Index: 299

Text #9

Max Range: 0.1113. Min Range: -0.1113

Max Act: 0.0937. Min Act: -0.0002

Data Index: 340935 (C4 (Web Text))

Max Activating Token Index: 370

Text #10

Max Range: 0.1113. Min Range: -0.1113

Max Act: 0.0960. Min Act: -0.0002

Data Index: 730016 (C4 (Web Text))

Max Activating Token Index: 182

Text #11

Max Range: 0.1113. Min Range: -0.1113

Max Act: 0.0888. Min Act: -0.0002

Data Index: 373829 (C4 (Web Text))

Max Activating Token Index: 864

Text #12

Max Range: 0.1113. Min Range: -0.1113

Max Act: 0.0925. Min Act: -0.0002

Data Index: 677480 (C4 (Web Text))

Max Activating Token Index: 180

Text #13

Max Range: 0.1113. Min Range: -0.1113

Max Act: 0.0789. Min Act: -0.0002

Data Index: 155059 (C4 (Web Text))

Max Activating Token Index: 637

Text #14

Max Range: 0.1113. Min Range: -0.1113

Max Act: 0.0918. Min Act: -0.0002

Data Index: 321939 (C4 (Web Text))

Max Activating Token Index: 706

Text #15

Max Range: 0.1113. Min Range: -0.1113

Max Act: 0.0769. Min Act: -0.0002

Data Index: 1285970 (C4 (Web Text))

Max Activating Token Index: 238

Text #16

Max Range: 0.1113. Min Range: -0.1113

Max Act: 0.0905. Min Act: -0.0002

Data Index: 244579 (C4 (Web Text))

Max Activating Token Index: 291

Text #17

Max Range: 0.1113. Min Range: -0.1113

Max Act: 0.0867. Min Act: -0.0002

Data Index: 987309 (C4 (Web Text))

Max Activating Token Index: 670

Text #18

Max Range: 0.1113. Min Range: -0.1113

Max Act: 0.0895. Min Act: -0.0002

Data Index: 216830 (C4 (Web Text))

Max Activating Token Index: 166

Text #19

Max Range: 0.1113. Min Range: -0.1113

Max Act: 0.0831. Min Act: -0.0002

Data Index: 75973 (C4 (Web Text))

Max Activating Token Index: 750