Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2222 in Layer 1

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.9895. Min Range: -0.9895

Max Act: 0.9668. Min Act: -0.0003

Data Index: 911343 (C4 (Web Text))

Max Activating Token Index: 854

Text #1

Max Range: 0.9895. Min Range: -0.9895

Max Act: 0.9895. Min Act: -0.0003

Data Index: 469433 (C4 (Web Text))

Max Activating Token Index: 695

Text #2

Max Range: 0.9895. Min Range: -0.9895

Max Act: 0.9189. Min Act: -0.0003

Data Index: 169464 (C4 (Web Text))

Max Activating Token Index: 622

Text #3

Max Range: 0.9895. Min Range: -0.9895

Max Act: 0.9402. Min Act: -0.0003

Data Index: 948808 (C4 (Web Text))

Max Activating Token Index: 50

Text #4

Max Range: 0.9895. Min Range: -0.9895

Max Act: 0.9278. Min Act: -0.0003

Data Index: 995399 (C4 (Web Text))

Max Activating Token Index: 347

Text #5

Max Range: 0.9895. Min Range: -0.9895

Max Act: 0.8947. Min Act: -0.0003

Data Index: 764300 (C4 (Web Text))

Max Activating Token Index: 540

Text #6

Max Range: 0.9895. Min Range: -0.9895

Max Act: 0.8773. Min Act: -0.0003

Data Index: 1210492 (C4 (Web Text))

Max Activating Token Index: 622

Text #7

Max Range: 0.9895. Min Range: -0.9895

Max Act: 0.8420. Min Act: -0.0003

Data Index: 635454 (C4 (Web Text))

Max Activating Token Index: 970

Text #8

Max Range: 0.9895. Min Range: -0.9895

Max Act: 0.8398. Min Act: -0.0003

Data Index: 378230 (C4 (Web Text))

Max Activating Token Index: 854

Text #9

Max Range: 0.9895. Min Range: -0.9895

Max Act: 0.8530. Min Act: -0.0003

Data Index: 1271205 (C4 (Web Text))

Max Activating Token Index: 2

Text #10

Max Range: 0.9895. Min Range: -0.9895

Max Act: 0.8494. Min Act: -0.0003

Data Index: 186582 (C4 (Web Text))

Max Activating Token Index: 33

Text #11

Max Range: 0.9895. Min Range: -0.9895

Max Act: 0.8496. Min Act: -0.0003

Data Index: 12738 (C4 (Web Text))

Max Activating Token Index: 519

Text #12

Max Range: 0.9895. Min Range: -0.9895

Max Act: 0.8276. Min Act: -0.0003

Data Index: 1127445 (C4 (Web Text))

Max Activating Token Index: 760

Text #13

Max Range: 0.9895. Min Range: -0.9895

Max Act: 0.8134. Min Act: -0.0003

Data Index: 494820 (C4 (Web Text))

Max Activating Token Index: 130

Text #14

Max Range: 0.9895. Min Range: -0.9895

Max Act: 0.8398. Min Act: -0.0003

Data Index: 544459 (C4 (Web Text))

Max Activating Token Index: 106

Text #15

Max Range: 0.9895. Min Range: -0.9895

Max Act: 0.8397. Min Act: -0.0003

Data Index: 225667 (C4 (Web Text))

Max Activating Token Index: 73

Text #16

Max Range: 0.9895. Min Range: -0.9895

Max Act: 0.8357. Min Act: -0.0003

Data Index: 98705 (C4 (Web Text))

Max Activating Token Index: 63

Text #17

Max Range: 0.9895. Min Range: -0.9895

Max Act: 0.8571. Min Act: -0.0003

Data Index: 693344 (C4 (Web Text))

Max Activating Token Index: 807

Text #18

Max Range: 0.9895. Min Range: -0.9895

Max Act: 0.8258. Min Act: -0.0003

Data Index: 357341 (C4 (Web Text))

Max Activating Token Index: 22

Text #19

Max Range: 0.9895. Min Range: -0.9895

Max Act: 0.8193. Min Act: -0.0003

Data Index: 612458 (C4 (Web Text))

Max Activating Token Index: 729