Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3886 in Layer 10

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 0.9590. Min Range: -0.9590

Max Act: 0.9590. Min Act: -0.0000

Data Index: 250748 (C4 (Web Text))

Max Activating Token Index: 957

Text #1

Max Range: 0.9590. Min Range: -0.9590

Max Act: 0.9442. Min Act: -0.0000

Data Index: 1165291 (C4 (Web Text))

Max Activating Token Index: 709

Text #2

Max Range: 0.9590. Min Range: -0.9590

Max Act: 0.9364. Min Act: -0.0000

Data Index: 671250 (C4 (Web Text))

Max Activating Token Index: 1003

Text #3

Max Range: 0.9590. Min Range: -0.9590

Max Act: 0.9190. Min Act: -0.0000

Data Index: 48277 (C4 (Web Text))

Max Activating Token Index: 594

Text #4

Max Range: 0.9590. Min Range: -0.9590

Max Act: 0.8926. Min Act: -0.0000

Data Index: 1244246 (C4 (Web Text))

Max Activating Token Index: 318

Text #5

Max Range: 0.9590. Min Range: -0.9590

Max Act: 0.8776. Min Act: -0.0000

Data Index: 1348393 (C4 (Web Text))

Max Activating Token Index: 421

Text #6

Max Range: 0.9590. Min Range: -0.9590

Max Act: 0.9172. Min Act: -0.0000

Data Index: 355072 (C4 (Web Text))

Max Activating Token Index: 461

Text #7

Max Range: 0.9590. Min Range: -0.9590

Max Act: 0.8974. Min Act: -0.0000

Data Index: 310792 (C4 (Web Text))

Max Activating Token Index: 529

Text #8

Max Range: 0.9590. Min Range: -0.9590

Max Act: 0.8627. Min Act: -0.0000

Data Index: 435461 (C4 (Web Text))

Max Activating Token Index: 664

Text #9

Max Range: 0.9590. Min Range: -0.9590

Max Act: 0.8867. Min Act: -0.0000

Data Index: 524868 (C4 (Web Text))

Max Activating Token Index: 45

Text #10

Max Range: 0.9590. Min Range: -0.9590

Max Act: 0.8815. Min Act: -0.0000

Data Index: 390135 (C4 (Web Text))

Max Activating Token Index: 196

Text #11

Max Range: 0.9590. Min Range: -0.9590

Max Act: 0.8483. Min Act: -0.0000

Data Index: 1276923 (C4 (Web Text))

Max Activating Token Index: 409

Text #12

Max Range: 0.9590. Min Range: -0.9590

Max Act: 0.8384. Min Act: -0.0000

Data Index: 76735 (C4 (Web Text))

Max Activating Token Index: 511

Text #13

Max Range: 0.9590. Min Range: -0.9590

Max Act: 0.8452. Min Act: -0.0000

Data Index: 802642 (C4 (Web Text))

Max Activating Token Index: 625

Text #14

Max Range: 0.9590. Min Range: -0.9590

Max Act: 0.8662. Min Act: -0.0000

Data Index: 908388 (C4 (Web Text))

Max Activating Token Index: 375

Text #15

Max Range: 0.9590. Min Range: -0.9590

Max Act: 0.8435. Min Act: -0.0000

Data Index: 1250581 (C4 (Web Text))

Max Activating Token Index: 710

Text #16

Max Range: 0.9590. Min Range: -0.9590

Max Act: 0.8735. Min Act: -0.0000

Data Index: 844750 (C4 (Web Text))

Max Activating Token Index: 20

Text #17

Max Range: 0.9590. Min Range: -0.9590

Max Act: 0.8127. Min Act: -0.0000

Data Index: 378285 (C4 (Web Text))

Max Activating Token Index: 361

Text #18

Max Range: 0.9590. Min Range: -0.9590

Max Act: 0.8276. Min Act: -0.0000

Data Index: 590649 (C4 (Web Text))

Max Activating Token Index: 496

Text #19

Max Range: 0.9590. Min Range: -0.9590

Max Act: 0.8162. Min Act: -0.0000

Data Index: 322328 (C4 (Web Text))

Max Activating Token Index: 401