Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4561 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.1367. Min Range: -0.1367

Max Act: 0.1118. Min Act: -0.0002

Data Index: 752214 (C4 (Web Text))

Max Activating Token Index: 677

Text #1

Max Range: 0.1367. Min Range: -0.1367

Max Act: 0.1295. Min Act: -0.0002

Data Index: 457533 (C4 (Web Text))

Max Activating Token Index: 90

Text #2

Max Range: 0.1367. Min Range: -0.1367

Max Act: 0.1367. Min Act: -0.0002

Data Index: 690823 (C4 (Web Text))

Max Activating Token Index: 135

Text #3

Max Range: 0.1367. Min Range: -0.1367

Max Act: 0.1116. Min Act: -0.0002

Data Index: 846044 (C4 (Web Text))

Max Activating Token Index: 160

Text #4

Max Range: 0.1367. Min Range: -0.1367

Max Act: 0.1188. Min Act: -0.0002

Data Index: 1298275 (C4 (Web Text))

Max Activating Token Index: 856

Text #5

Max Range: 0.1367. Min Range: -0.1367

Max Act: 0.1090. Min Act: -0.0002

Data Index: 1044363 (C4 (Web Text))

Max Activating Token Index: 129

Text #6

Max Range: 0.1367. Min Range: -0.1367

Max Act: 0.1081. Min Act: -0.0002

Data Index: 339077 (C4 (Web Text))

Max Activating Token Index: 327

Text #7

Max Range: 0.1367. Min Range: -0.1367

Max Act: 0.1195. Min Act: -0.0002

Data Index: 942489 (C4 (Web Text))

Max Activating Token Index: 243

Text #8

Max Range: 0.1367. Min Range: -0.1367

Max Act: 0.0982. Min Act: -0.0002

Data Index: 899838 (C4 (Web Text))

Max Activating Token Index: 293

Text #9

Max Range: 0.1367. Min Range: -0.1367

Max Act: 0.1067. Min Act: -0.0002

Data Index: 931061 (C4 (Web Text))

Max Activating Token Index: 492

Text #10

Max Range: 0.1367. Min Range: -0.1367

Max Act: 0.1012. Min Act: -0.0002

Data Index: 868608 (C4 (Web Text))

Max Activating Token Index: 119

Text #11

Max Range: 0.1367. Min Range: -0.1367

Max Act: 0.1024. Min Act: -0.0002

Data Index: 748417 (C4 (Web Text))

Max Activating Token Index: 9

Text #12

Max Range: 0.1367. Min Range: -0.1367

Max Act: 0.1053. Min Act: -0.0002

Data Index: 721404 (C4 (Web Text))

Max Activating Token Index: 547

Text #13

Max Range: 0.1367. Min Range: -0.1367

Max Act: 0.0943. Min Act: -0.0002

Data Index: 1203533 (C4 (Web Text))

Max Activating Token Index: 166

Text #14

Max Range: 0.1367. Min Range: -0.1367

Max Act: 0.0956. Min Act: -0.0002

Data Index: 958463 (C4 (Web Text))

Max Activating Token Index: 950

Text #15

Max Range: 0.1367. Min Range: -0.1367

Max Act: 0.1045. Min Act: -0.0002

Data Index: 617832 (C4 (Web Text))

Max Activating Token Index: 729

Text #16

Max Range: 0.1367. Min Range: -0.1367

Max Act: 0.1127. Min Act: -0.0002

Data Index: 510665 (C4 (Web Text))

Max Activating Token Index: 158

Text #17

Max Range: 0.1367. Min Range: -0.1367

Max Act: 0.1050. Min Act: -0.0002

Data Index: 589382 (C4 (Web Text))

Max Activating Token Index: 969

Text #18

Max Range: 0.1367. Min Range: -0.1367

Max Act: 0.1090. Min Act: -0.0002

Data Index: 919423 (C4 (Web Text))

Max Activating Token Index: 802

Text #19

Max Range: 0.1367. Min Range: -0.1367

Max Act: 0.1064. Min Act: -0.0002

Data Index: 207974 (C4 (Web Text))

Max Activating Token Index: 349