Model: SoLU Model: 6 Layers, 3072 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2227 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-6l')

Text #0

Max Range: 0.1131. Min Range: -0.1131

Max Act: 0.1131. Min Act: -0.0001

Data Index: 1341506 (C4 (Web Text))

Max Activating Token Index: 496

Text #1

Max Range: 0.1131. Min Range: -0.1131

Max Act: 0.1027. Min Act: -0.0001

Data Index: 932608 (C4 (Web Text))

Max Activating Token Index: 117

Text #2

Max Range: 0.1131. Min Range: -0.1131

Max Act: 0.0908. Min Act: -0.0001

Data Index: 384751 (C4 (Web Text))

Max Activating Token Index: 206

Text #3

Max Range: 0.1131. Min Range: -0.1131

Max Act: 0.0773. Min Act: -0.0001

Data Index: 987958 (C4 (Web Text))

Max Activating Token Index: 692

Text #4

Max Range: 0.1131. Min Range: -0.1131

Max Act: 0.0795. Min Act: -0.0001

Data Index: 1337141 (C4 (Web Text))

Max Activating Token Index: 618

Text #5

Max Range: 0.1131. Min Range: -0.1131

Max Act: 0.0812. Min Act: -0.0001

Data Index: 280846 (C4 (Web Text))

Max Activating Token Index: 177

Text #6

Max Range: 0.1131. Min Range: -0.1131

Max Act: 0.0806. Min Act: -0.0001

Data Index: 148840 (C4 (Web Text))

Max Activating Token Index: 664

Text #7

Max Range: 0.1131. Min Range: -0.1131

Max Act: 0.0788. Min Act: -0.0001

Data Index: 897115 (C4 (Web Text))

Max Activating Token Index: 939

Text #8

Max Range: 0.1131. Min Range: -0.1131

Max Act: 0.0684. Min Act: -0.0001

Data Index: 736779 (C4 (Web Text))

Max Activating Token Index: 695

Text #9

Max Range: 0.1131. Min Range: -0.1131

Max Act: 0.0868. Min Act: -0.0001

Data Index: 1307075 (C4 (Web Text))

Max Activating Token Index: 781

Text #10

Max Range: 0.1131. Min Range: -0.1131

Max Act: 0.0723. Min Act: -0.0001

Data Index: 406108 (C4 (Web Text))

Max Activating Token Index: 340

Text #11

Max Range: 0.1131. Min Range: -0.1131

Max Act: 0.0702. Min Act: -0.0001

Data Index: 735411 (C4 (Web Text))

Max Activating Token Index: 356

Text #12

Max Range: 0.1131. Min Range: -0.1131

Max Act: 0.0785. Min Act: -0.0001

Data Index: 1337654 (C4 (Web Text))

Max Activating Token Index: 211

Text #13

Max Range: 0.1131. Min Range: -0.1131

Max Act: 0.0684. Min Act: -0.0001

Data Index: 1653686 (Python Code)

Max Activating Token Index: 883

Text #14

Max Range: 0.1131. Min Range: -0.1131

Max Act: 0.0702. Min Act: -0.0001

Data Index: 66605 (C4 (Web Text))

Max Activating Token Index: 570

Text #15

Max Range: 0.1131. Min Range: -0.1131

Max Act: 0.0634. Min Act: -0.0001

Data Index: 784095 (C4 (Web Text))

Max Activating Token Index: 928

Text #16

Max Range: 0.1131. Min Range: -0.1131

Max Act: 0.0650. Min Act: -0.0001

Data Index: 859490 (C4 (Web Text))

Max Activating Token Index: 215

Text #17

Max Range: 0.1131. Min Range: -0.1131

Max Act: 0.0845. Min Act: -0.0001

Data Index: 849106 (C4 (Web Text))

Max Activating Token Index: 955

Text #18

Max Range: 0.1131. Min Range: -0.1131

Max Act: 0.0668. Min Act: -0.0001

Data Index: 236954 (C4 (Web Text))

Max Activating Token Index: 529

Text #19

Max Range: 0.1131. Min Range: -0.1131

Max Act: 0.0567. Min Act: -0.0001

Data Index: 605025 (C4 (Web Text))

Max Activating Token Index: 630