Model: SoLU Model: 4 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1084 in Layer 3

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-4l')

Text #0

Max Range: 0.1119. Min Range: -0.1119

Max Act: 0.1110. Min Act: -0.0002

Data Index: 467903 (C4 (Web Text))

Max Activating Token Index: 868

Text #1

Max Range: 0.1119. Min Range: -0.1119

Max Act: 0.1119. Min Act: -0.0002

Data Index: 207461 (C4 (Web Text))

Max Activating Token Index: 679

Text #2

Max Range: 0.1119. Min Range: -0.1119

Max Act: 0.1083. Min Act: -0.0002

Data Index: 620989 (C4 (Web Text))

Max Activating Token Index: 310

Text #3

Max Range: 0.1119. Min Range: -0.1119

Max Act: 0.1114. Min Act: -0.0002

Data Index: 117976 (C4 (Web Text))

Max Activating Token Index: 789

Text #4

Max Range: 0.1119. Min Range: -0.1119

Max Act: 0.1056. Min Act: -0.0002

Data Index: 21076 (C4 (Web Text))

Max Activating Token Index: 634

Text #5

Max Range: 0.1119. Min Range: -0.1119

Max Act: 0.1015. Min Act: -0.0002

Data Index: 433850 (C4 (Web Text))

Max Activating Token Index: 136

Text #6

Max Range: 0.1119. Min Range: -0.1119

Max Act: 0.1003. Min Act: -0.0002

Data Index: 820824 (C4 (Web Text))

Max Activating Token Index: 907

Text #7

Max Range: 0.1119. Min Range: -0.1119

Max Act: 0.1006. Min Act: -0.0002

Data Index: 242940 (C4 (Web Text))

Max Activating Token Index: 160

Text #8

Max Range: 0.1119. Min Range: -0.1119

Max Act: 0.0995. Min Act: -0.0002

Data Index: 378101 (C4 (Web Text))

Max Activating Token Index: 105

Text #9

Max Range: 0.1119. Min Range: -0.1119

Max Act: 0.1007. Min Act: -0.0002

Data Index: 133317 (C4 (Web Text))

Max Activating Token Index: 469

Text #10

Max Range: 0.1119. Min Range: -0.1119

Max Act: 0.0975. Min Act: -0.0002

Data Index: 455730 (C4 (Web Text))

Max Activating Token Index: 908

Text #11

Max Range: 0.1119. Min Range: -0.1119

Max Act: 0.0934. Min Act: -0.0002

Data Index: 553541 (C4 (Web Text))

Max Activating Token Index: 636

Text #12

Max Range: 0.1119. Min Range: -0.1119

Max Act: 0.0975. Min Act: -0.0002

Data Index: 1118326 (C4 (Web Text))

Max Activating Token Index: 98

Text #13

Max Range: 0.1119. Min Range: -0.1119

Max Act: 0.0994. Min Act: -0.0002

Data Index: 923833 (C4 (Web Text))

Max Activating Token Index: 123

Text #14

Max Range: 0.1119. Min Range: -0.1119

Max Act: 0.0970. Min Act: -0.0002

Data Index: 64140 (C4 (Web Text))

Max Activating Token Index: 12

Text #15

Max Range: 0.1119. Min Range: -0.1119

Max Act: 0.0977. Min Act: -0.0002

Data Index: 118328 (C4 (Web Text))

Max Activating Token Index: 169

Text #16

Max Range: 0.1119. Min Range: -0.1119

Max Act: 0.0949. Min Act: -0.0002

Data Index: 176909 (C4 (Web Text))

Max Activating Token Index: 950

Text #17

Max Range: 0.1119. Min Range: -0.1119

Max Act: 0.0950. Min Act: -0.0002

Data Index: 1122166 (C4 (Web Text))

Max Activating Token Index: 294

Text #18

Max Range: 0.1119. Min Range: -0.1119

Max Act: 0.0974. Min Act: -0.0002

Data Index: 1128220 (C4 (Web Text))

Max Activating Token Index: 485

Text #19

Max Range: 0.1119. Min Range: -0.1119

Max Act: 0.0952. Min Act: -0.0002

Data Index: 114196 (C4 (Web Text))

Max Activating Token Index: 9