Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3576 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 0.0827. Min Range: -0.0827

Max Act: 0.0824. Min Act: -0.0002

Data Index: 1190262 (C4 (Web Text))

Max Activating Token Index: 626

Text #1

Max Range: 0.0827. Min Range: -0.0827

Max Act: 0.0827. Min Act: -0.0002

Data Index: 280763 (C4 (Web Text))

Max Activating Token Index: 403

Text #2

Max Range: 0.0827. Min Range: -0.0827

Max Act: 0.0788. Min Act: -0.0002

Data Index: 638114 (C4 (Web Text))

Max Activating Token Index: 574

Text #3

Max Range: 0.0827. Min Range: -0.0827

Max Act: 0.0791. Min Act: -0.0002

Data Index: 1039272 (C4 (Web Text))

Max Activating Token Index: 35

Text #4

Max Range: 0.0827. Min Range: -0.0827

Max Act: 0.0782. Min Act: -0.0002

Data Index: 427599 (C4 (Web Text))

Max Activating Token Index: 111

Text #5

Max Range: 0.0827. Min Range: -0.0827

Max Act: 0.0747. Min Act: -0.0002

Data Index: 318358 (C4 (Web Text))

Max Activating Token Index: 891

Text #6

Max Range: 0.0827. Min Range: -0.0827

Max Act: 0.0711. Min Act: -0.0002

Data Index: 1094840 (C4 (Web Text))

Max Activating Token Index: 494

Text #7

Max Range: 0.0827. Min Range: -0.0827

Max Act: 0.0731. Min Act: -0.0002

Data Index: 624339 (C4 (Web Text))

Max Activating Token Index: 906

Text #8

Max Range: 0.0827. Min Range: -0.0827

Max Act: 0.0710. Min Act: -0.0002

Data Index: 390926 (C4 (Web Text))

Max Activating Token Index: 796

Text #9

Max Range: 0.0827. Min Range: -0.0827

Max Act: 0.0684. Min Act: -0.0002

Data Index: 997233 (C4 (Web Text))

Max Activating Token Index: 646

Text #10

Max Range: 0.0827. Min Range: -0.0827

Max Act: 0.0651. Min Act: -0.0002

Data Index: 311938 (C4 (Web Text))

Max Activating Token Index: 495

Text #11

Max Range: 0.0827. Min Range: -0.0827

Max Act: 0.0666. Min Act: -0.0002

Data Index: 1138302 (C4 (Web Text))

Max Activating Token Index: 847

Text #12

Max Range: 0.0827. Min Range: -0.0827

Max Act: 0.0625. Min Act: -0.0002

Data Index: 541996 (C4 (Web Text))

Max Activating Token Index: 593

Text #13

Max Range: 0.0827. Min Range: -0.0827

Max Act: 0.0641. Min Act: -0.0002

Data Index: 895443 (C4 (Web Text))

Max Activating Token Index: 363

Text #14

Max Range: 0.0827. Min Range: -0.0827

Max Act: 0.0629. Min Act: -0.0002

Data Index: 120001 (C4 (Web Text))

Max Activating Token Index: 889

Text #15

Max Range: 0.0827. Min Range: -0.0827

Max Act: 0.0583. Min Act: -0.0002

Data Index: 551109 (C4 (Web Text))

Max Activating Token Index: 320

Text #16

Max Range: 0.0827. Min Range: -0.0827

Max Act: 0.0623. Min Act: -0.0002

Data Index: 517549 (C4 (Web Text))

Max Activating Token Index: 923

Text #17

Max Range: 0.0827. Min Range: -0.0827

Max Act: 0.0600. Min Act: -0.0002

Data Index: 1275152 (C4 (Web Text))

Max Activating Token Index: 813

Text #18

Max Range: 0.0827. Min Range: -0.0827

Max Act: 0.0621. Min Act: -0.0002

Data Index: 329980 (C4 (Web Text))

Max Activating Token Index: 782

Text #19

Max Range: 0.0827. Min Range: -0.0827

Max Act: 0.0558. Min Act: -0.0002

Data Index: 686121 (C4 (Web Text))

Max Activating Token Index: 1005