Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3870 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.0932. Min Range: -0.0932

Max Act: 0.0915. Min Act: -0.0002

Data Index: 322723 (C4 (Web Text))

Max Activating Token Index: 98

Text #1

Max Range: 0.0932. Min Range: -0.0932

Max Act: 0.0932. Min Act: -0.0002

Data Index: 313544 (C4 (Web Text))

Max Activating Token Index: 731

Text #2

Max Range: 0.0932. Min Range: -0.0932

Max Act: 0.0841. Min Act: -0.0002

Data Index: 57005 (C4 (Web Text))

Max Activating Token Index: 715

Text #3

Max Range: 0.0932. Min Range: -0.0932

Max Act: 0.0773. Min Act: -0.0002

Data Index: 1395191 (Python Code)

Max Activating Token Index: 594

Text #4

Max Range: 0.0932. Min Range: -0.0932

Max Act: 0.0708. Min Act: -0.0002

Data Index: 1607793 (Python Code)

Max Activating Token Index: 669

Text #5

Max Range: 0.0932. Min Range: -0.0932

Max Act: 0.0709. Min Act: -0.0002

Data Index: 1620354 (Python Code)

Max Activating Token Index: 593

Text #6

Max Range: 0.0932. Min Range: -0.0932

Max Act: 0.0806. Min Act: -0.0002

Data Index: 44202 (C4 (Web Text))

Max Activating Token Index: 651

Text #7

Max Range: 0.0932. Min Range: -0.0932

Max Act: 0.0697. Min Act: -0.0002

Data Index: 88426 (C4 (Web Text))

Max Activating Token Index: 487

Text #8

Max Range: 0.0932. Min Range: -0.0932

Max Act: 0.0669. Min Act: -0.0002

Data Index: 1214447 (C4 (Web Text))

Max Activating Token Index: 776

Text #9

Max Range: 0.0932. Min Range: -0.0932

Max Act: 0.0732. Min Act: -0.0002

Data Index: 951766 (C4 (Web Text))

Max Activating Token Index: 792

Text #10

Max Range: 0.0932. Min Range: -0.0932

Max Act: 0.0716. Min Act: -0.0002

Data Index: 502611 (C4 (Web Text))

Max Activating Token Index: 458

Text #11

Max Range: 0.0932. Min Range: -0.0932

Max Act: 0.0720. Min Act: -0.0002

Data Index: 958715 (C4 (Web Text))

Max Activating Token Index: 240

Text #12

Max Range: 0.0932. Min Range: -0.0932

Max Act: 0.0738. Min Act: -0.0002

Data Index: 1471996 (Python Code)

Max Activating Token Index: 186

Text #13

Max Range: 0.0932. Min Range: -0.0932

Max Act: 0.0711. Min Act: -0.0002

Data Index: 1334961 (C4 (Web Text))

Max Activating Token Index: 864

Text #14

Max Range: 0.0932. Min Range: -0.0932

Max Act: 0.0747. Min Act: -0.0002

Data Index: 1092192 (C4 (Web Text))

Max Activating Token Index: 162

Text #15

Max Range: 0.0932. Min Range: -0.0932

Max Act: 0.0757. Min Act: -0.0002

Data Index: 1494296 (Python Code)

Max Activating Token Index: 873

Text #16

Max Range: 0.0932. Min Range: -0.0932

Max Act: 0.0731. Min Act: -0.0002

Data Index: 955918 (C4 (Web Text))

Max Activating Token Index: 893

Text #17

Max Range: 0.0932. Min Range: -0.0932

Max Act: 0.0735. Min Act: -0.0002

Data Index: 1504425 (Python Code)

Max Activating Token Index: 179

Text #18

Max Range: 0.0932. Min Range: -0.0932

Max Act: 0.0679. Min Act: -0.0002

Data Index: 526643 (C4 (Web Text))

Max Activating Token Index: 59

Text #19

Max Range: 0.0932. Min Range: -0.0932

Max Act: 0.0709. Min Act: -0.0002

Data Index: 832284 (C4 (Web Text))

Max Activating Token Index: 394