Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4854 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 2.0187. Min Range: -2.0187

Max Act: 2.0187. Min Act: -0.0000

Data Index: 1251460 (C4 (Web Text))

Max Activating Token Index: 335

Text #1

Max Range: 2.0187. Min Range: -2.0187

Max Act: 1.8817. Min Act: -0.0000

Data Index: 954268 (C4 (Web Text))

Max Activating Token Index: 396

Text #2

Max Range: 2.0187. Min Range: -2.0187

Max Act: 1.8809. Min Act: -0.0000

Data Index: 1271001 (C4 (Web Text))

Max Activating Token Index: 947

Text #3

Max Range: 2.0187. Min Range: -2.0187

Max Act: 1.8636. Min Act: -0.0000

Data Index: 1298556 (C4 (Web Text))

Max Activating Token Index: 515

Text #4

Max Range: 2.0187. Min Range: -2.0187

Max Act: 1.8387. Min Act: -0.0000

Data Index: 1216225 (C4 (Web Text))

Max Activating Token Index: 68

Text #5

Max Range: 2.0187. Min Range: -2.0187

Max Act: 1.7601. Min Act: -0.0000

Data Index: 1219585 (C4 (Web Text))

Max Activating Token Index: 772

Text #6

Max Range: 2.0187. Min Range: -2.0187

Max Act: 1.6940. Min Act: -0.0000

Data Index: 1196247 (C4 (Web Text))

Max Activating Token Index: 474

Text #7

Max Range: 2.0187. Min Range: -2.0187

Max Act: 1.5376. Min Act: -0.0000

Data Index: 740230 (C4 (Web Text))

Max Activating Token Index: 422

Text #8

Max Range: 2.0187. Min Range: -2.0187

Max Act: 1.6242. Min Act: -0.0000

Data Index: 215720 (C4 (Web Text))

Max Activating Token Index: 363

Text #9

Max Range: 2.0187. Min Range: -2.0187

Max Act: 1.7129. Min Act: -0.0000

Data Index: 370075 (C4 (Web Text))

Max Activating Token Index: 48

Text #10

Max Range: 2.0187. Min Range: -2.0187

Max Act: 1.7083. Min Act: -0.0000

Data Index: 1314426 (C4 (Web Text))

Max Activating Token Index: 803

Text #11

Max Range: 2.0187. Min Range: -2.0187

Max Act: 1.5989. Min Act: -0.0000

Data Index: 340396 (C4 (Web Text))

Max Activating Token Index: 274

Text #12

Max Range: 2.0187. Min Range: -2.0187

Max Act: 1.5079. Min Act: -0.0000

Data Index: 1098164 (C4 (Web Text))

Max Activating Token Index: 110

Text #13

Max Range: 2.0187. Min Range: -2.0187

Max Act: 1.5609. Min Act: -0.0000

Data Index: 247037 (C4 (Web Text))

Max Activating Token Index: 35

Text #14

Max Range: 2.0187. Min Range: -2.0187

Max Act: 1.4687. Min Act: -0.0000

Data Index: 986249 (C4 (Web Text))

Max Activating Token Index: 982

Text #15

Max Range: 2.0187. Min Range: -2.0187

Max Act: 1.5170. Min Act: -0.0000

Data Index: 66593 (C4 (Web Text))

Max Activating Token Index: 597

Text #16

Max Range: 2.0187. Min Range: -2.0187

Max Act: 1.5227. Min Act: -0.0000

Data Index: 186381 (C4 (Web Text))

Max Activating Token Index: 497

Text #17

Max Range: 2.0187. Min Range: -2.0187

Max Act: 1.6115. Min Act: -0.0000

Data Index: 679454 (C4 (Web Text))

Max Activating Token Index: 13

Text #18

Max Range: 2.0187. Min Range: -2.0187

Max Act: 1.4312. Min Act: -0.0000

Data Index: 1118701 (C4 (Web Text))

Max Activating Token Index: 350

Text #19

Max Range: 2.0187. Min Range: -2.0187

Max Act: 1.5261. Min Act: -0.0000

Data Index: 504902 (C4 (Web Text))

Max Activating Token Index: 33