Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2951 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.0434. Min Range: -1.0434

Max Act: 1.0183. Min Act: -0.0000

Data Index: 434795 (C4 (Web Text))

Max Activating Token Index: 314

Text #1

Max Range: 1.0434. Min Range: -1.0434

Max Act: 1.0111. Min Act: -0.0000

Data Index: 242125 (C4 (Web Text))

Max Activating Token Index: 383

Text #2

Max Range: 1.0434. Min Range: -1.0434

Max Act: 1.0434. Min Act: -0.0000

Data Index: 1012183 (C4 (Web Text))

Max Activating Token Index: 70

Text #3

Max Range: 1.0434. Min Range: -1.0434

Max Act: 1.0336. Min Act: -0.0000

Data Index: 95220 (C4 (Web Text))

Max Activating Token Index: 506

Text #4

Max Range: 1.0434. Min Range: -1.0434

Max Act: 0.9994. Min Act: -0.0000

Data Index: 526136 (C4 (Web Text))

Max Activating Token Index: 99

Text #5

Max Range: 1.0434. Min Range: -1.0434

Max Act: 1.0220. Min Act: -0.0000

Data Index: 863180 (C4 (Web Text))

Max Activating Token Index: 293

Text #6

Max Range: 1.0434. Min Range: -1.0434

Max Act: 1.0213. Min Act: -0.0000

Data Index: 965516 (C4 (Web Text))

Max Activating Token Index: 178

Text #7

Max Range: 1.0434. Min Range: -1.0434

Max Act: 0.9848. Min Act: -0.0000

Data Index: 964810 (C4 (Web Text))

Max Activating Token Index: 364

Text #8

Max Range: 1.0434. Min Range: -1.0434

Max Act: 1.0135. Min Act: -0.0000

Data Index: 100449 (C4 (Web Text))

Max Activating Token Index: 361

Text #9

Max Range: 1.0434. Min Range: -1.0434

Max Act: 1.0125. Min Act: -0.0000

Data Index: 391358 (C4 (Web Text))

Max Activating Token Index: 241

Text #10

Max Range: 1.0434. Min Range: -1.0434

Max Act: 1.0056. Min Act: -0.0000

Data Index: 1277888 (C4 (Web Text))

Max Activating Token Index: 53

Text #11

Max Range: 1.0434. Min Range: -1.0434

Max Act: 0.9747. Min Act: -0.0000

Data Index: 1198181 (C4 (Web Text))

Max Activating Token Index: 145

Text #12

Max Range: 1.0434. Min Range: -1.0434

Max Act: 1.0321. Min Act: -0.0000

Data Index: 413163 (C4 (Web Text))

Max Activating Token Index: 839

Text #13

Max Range: 1.0434. Min Range: -1.0434

Max Act: 0.9365. Min Act: -0.0000

Data Index: 346435 (C4 (Web Text))

Max Activating Token Index: 246

Text #14

Max Range: 1.0434. Min Range: -1.0434

Max Act: 0.9587. Min Act: -0.0000

Data Index: 29566 (C4 (Web Text))

Max Activating Token Index: 132

Text #15

Max Range: 1.0434. Min Range: -1.0434

Max Act: 0.9903. Min Act: -0.0000

Data Index: 887535 (C4 (Web Text))

Max Activating Token Index: 380

Text #16

Max Range: 1.0434. Min Range: -1.0434

Max Act: 0.9932. Min Act: -0.0000

Data Index: 1082181 (C4 (Web Text))

Max Activating Token Index: 286

Text #17

Max Range: 1.0434. Min Range: -1.0434

Max Act: 0.9612. Min Act: -0.0000

Data Index: 1236325 (C4 (Web Text))

Max Activating Token Index: 223

Text #18

Max Range: 1.0434. Min Range: -1.0434

Max Act: 0.9769. Min Act: -0.0000

Data Index: 1315561 (C4 (Web Text))

Max Activating Token Index: 88

Text #19

Max Range: 1.0434. Min Range: -1.0434

Max Act: 0.9751. Min Act: -0.0000

Data Index: 912134 (C4 (Web Text))

Max Activating Token Index: 591