Model: SoLU Model: 2 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 979 in Layer 1

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-2l')

Text #0

Max Range: 0.0865. Min Range: -0.0865

Max Act: 0.0844. Min Act: -0.0003

Data Index: 1027041 (C4 (Web Text))

Max Activating Token Index: 308

Text #1

Max Range: 0.0865. Min Range: -0.0865

Max Act: 0.0865. Min Act: -0.0003

Data Index: 1331087 (C4 (Web Text))

Max Activating Token Index: 323

Text #2

Max Range: 0.0865. Min Range: -0.0865

Max Act: 0.0849. Min Act: -0.0003

Data Index: 1233353 (C4 (Web Text))

Max Activating Token Index: 792

Text #3

Max Range: 0.0865. Min Range: -0.0865

Max Act: 0.0826. Min Act: -0.0003

Data Index: 1220666 (C4 (Web Text))

Max Activating Token Index: 154

Text #4

Max Range: 0.0865. Min Range: -0.0865

Max Act: 0.0834. Min Act: -0.0003

Data Index: 100368 (C4 (Web Text))

Max Activating Token Index: 163

Text #5

Max Range: 0.0865. Min Range: -0.0865

Max Act: 0.0811. Min Act: -0.0003

Data Index: 947590 (C4 (Web Text))

Max Activating Token Index: 167

Text #6

Max Range: 0.0865. Min Range: -0.0865

Max Act: 0.0795. Min Act: -0.0003

Data Index: 102522 (C4 (Web Text))

Max Activating Token Index: 886

Text #7

Max Range: 0.0865. Min Range: -0.0865

Max Act: 0.0803. Min Act: -0.0003

Data Index: 941763 (C4 (Web Text))

Max Activating Token Index: 156

Text #8

Max Range: 0.0865. Min Range: -0.0865

Max Act: 0.0801. Min Act: -0.0003

Data Index: 911337 (C4 (Web Text))

Max Activating Token Index: 514

Text #9

Max Range: 0.0865. Min Range: -0.0865

Max Act: 0.0812. Min Act: -0.0003

Data Index: 170455 (C4 (Web Text))

Max Activating Token Index: 604

Text #10

Max Range: 0.0865. Min Range: -0.0865

Max Act: 0.0790. Min Act: -0.0003

Data Index: 671810 (C4 (Web Text))

Max Activating Token Index: 533

Text #11

Max Range: 0.0865. Min Range: -0.0865

Max Act: 0.0763. Min Act: -0.0003

Data Index: 438107 (C4 (Web Text))

Max Activating Token Index: 864

Text #12

Max Range: 0.0865. Min Range: -0.0865

Max Act: 0.0780. Min Act: -0.0002

Data Index: 566460 (C4 (Web Text))

Max Activating Token Index: 96

Text #13

Max Range: 0.0865. Min Range: -0.0865

Max Act: 0.0750. Min Act: -0.0003

Data Index: 304845 (C4 (Web Text))

Max Activating Token Index: 830

Text #14

Max Range: 0.0865. Min Range: -0.0865

Max Act: 0.0822. Min Act: -0.0003

Data Index: 1236955 (C4 (Web Text))

Max Activating Token Index: 844

Text #15

Max Range: 0.0865. Min Range: -0.0865

Max Act: 0.0754. Min Act: -0.0003

Data Index: 275980 (C4 (Web Text))

Max Activating Token Index: 477

Text #16

Max Range: 0.0865. Min Range: -0.0865

Max Act: 0.0756. Min Act: -0.0003

Data Index: 262588 (C4 (Web Text))

Max Activating Token Index: 631

Text #17

Max Range: 0.0865. Min Range: -0.0865

Max Act: 0.0753. Min Act: -0.0003

Data Index: 819755 (C4 (Web Text))

Max Activating Token Index: 810

Text #18

Max Range: 0.0865. Min Range: -0.0865

Max Act: 0.0768. Min Act: -0.0003

Data Index: 633224 (C4 (Web Text))

Max Activating Token Index: 63

Text #19

Max Range: 0.0865. Min Range: -0.0865

Max Act: 0.0766. Min Act: -0.0004

Data Index: 738697 (C4 (Web Text))

Max Activating Token Index: 540