Model: SoLU Model: 4 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 202 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-4l')

Text #0

Max Range: 0.1298. Min Range: -0.1298

Max Act: 0.1220. Min Act: -0.0005

Data Index: 496179 (C4 (Web Text))

Max Activating Token Index: 61

Text #1

Max Range: 0.1298. Min Range: -0.1298

Max Act: 0.1124. Min Act: -0.0005

Data Index: 1074602 (C4 (Web Text))

Max Activating Token Index: 869

Text #2

Max Range: 0.1298. Min Range: -0.1298

Max Act: 0.1298. Min Act: -0.0004

Data Index: 544313 (C4 (Web Text))

Max Activating Token Index: 984

Text #3

Max Range: 0.1298. Min Range: -0.1298

Max Act: 0.1187. Min Act: -0.0005

Data Index: 1246442 (C4 (Web Text))

Max Activating Token Index: 772

Text #4

Max Range: 0.1298. Min Range: -0.1298

Max Act: 0.1147. Min Act: -0.0005

Data Index: 114894 (C4 (Web Text))

Max Activating Token Index: 490

Text #5

Max Range: 0.1298. Min Range: -0.1298

Max Act: 0.1211. Min Act: -0.0005

Data Index: 1153369 (C4 (Web Text))

Max Activating Token Index: 733

Text #6

Max Range: 0.1298. Min Range: -0.1298

Max Act: 0.1161. Min Act: -0.0005

Data Index: 1103174 (C4 (Web Text))

Max Activating Token Index: 766

Text #7

Max Range: 0.1298. Min Range: -0.1298

Max Act: 0.1126. Min Act: -0.0005

Data Index: 1352896 (C4 (Web Text))

Max Activating Token Index: 540

Text #8

Max Range: 0.1298. Min Range: -0.1298

Max Act: 0.1082. Min Act: -0.0005

Data Index: 1227261 (C4 (Web Text))

Max Activating Token Index: 579

Text #9

Max Range: 0.1298. Min Range: -0.1298

Max Act: 0.1125. Min Act: -0.0005

Data Index: 722466 (C4 (Web Text))

Max Activating Token Index: 220

Text #10

Max Range: 0.1298. Min Range: -0.1298

Max Act: 0.1115. Min Act: -0.0005

Data Index: 1214134 (C4 (Web Text))

Max Activating Token Index: 191

Text #11

Max Range: 0.1298. Min Range: -0.1298

Max Act: 0.1158. Min Act: -0.0005

Data Index: 1240572 (C4 (Web Text))

Max Activating Token Index: 888

Text #12

Max Range: 0.1298. Min Range: -0.1298

Max Act: 0.1106. Min Act: -0.0005

Data Index: 1259237 (C4 (Web Text))

Max Activating Token Index: 626

Text #13

Max Range: 0.1298. Min Range: -0.1298

Max Act: 0.1146. Min Act: -0.0005

Data Index: 445701 (C4 (Web Text))

Max Activating Token Index: 685

Text #14

Max Range: 0.1298. Min Range: -0.1298

Max Act: 0.1103. Min Act: -0.0005

Data Index: 1181063 (C4 (Web Text))

Max Activating Token Index: 738

Text #15

Max Range: 0.1298. Min Range: -0.1298

Max Act: 0.1087. Min Act: -0.0005

Data Index: 552013 (C4 (Web Text))

Max Activating Token Index: 616

Text #16

Max Range: 0.1298. Min Range: -0.1298

Max Act: 0.1088. Min Act: -0.0005

Data Index: 89026 (C4 (Web Text))

Max Activating Token Index: 985

Text #17

Max Range: 0.1298. Min Range: -0.1298

Max Act: 0.1068. Min Act: -0.0005

Data Index: 477618 (C4 (Web Text))

Max Activating Token Index: 264

Text #18

Max Range: 0.1298. Min Range: -0.1298

Max Act: 0.0987. Min Act: -0.0005

Data Index: 947833 (C4 (Web Text))

Max Activating Token Index: 880

Text #19

Max Range: 0.1298. Min Range: -0.1298

Max Act: 0.1063. Min Act: -0.0005

Data Index: 1034254 (C4 (Web Text))

Max Activating Token Index: 635