Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2091 in Layer 3

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.0038. Min Range: -1.0038

Max Act: 1.0038. Min Act: -0.0001

Data Index: 692045 (C4 (Web Text))

Max Activating Token Index: 194

Text #1

Max Range: 1.0038. Min Range: -1.0038

Max Act: 0.9072. Min Act: -0.0001

Data Index: 965062 (C4 (Web Text))

Max Activating Token Index: 940

Text #2

Max Range: 1.0038. Min Range: -1.0038

Max Act: 0.9772. Min Act: -0.0001

Data Index: 1603 (C4 (Web Text))

Max Activating Token Index: 247

Text #3

Max Range: 1.0038. Min Range: -1.0038

Max Act: 0.9487. Min Act: -0.0001

Data Index: 1110081 (C4 (Web Text))

Max Activating Token Index: 909

Text #4

Max Range: 1.0038. Min Range: -1.0038

Max Act: 0.9168. Min Act: -0.0001

Data Index: 877542 (C4 (Web Text))

Max Activating Token Index: 492

Text #5

Max Range: 1.0038. Min Range: -1.0038

Max Act: 0.9603. Min Act: -0.0001

Data Index: 526676 (C4 (Web Text))

Max Activating Token Index: 219

Text #6

Max Range: 1.0038. Min Range: -1.0038

Max Act: 0.9667. Min Act: -0.0001

Data Index: 463653 (C4 (Web Text))

Max Activating Token Index: 214

Text #7

Max Range: 1.0038. Min Range: -1.0038

Max Act: 0.9064. Min Act: -0.0001

Data Index: 1056356 (C4 (Web Text))

Max Activating Token Index: 830

Text #8

Max Range: 1.0038. Min Range: -1.0038

Max Act: 0.8482. Min Act: -0.0001

Data Index: 438459 (C4 (Web Text))

Max Activating Token Index: 838

Text #9

Max Range: 1.0038. Min Range: -1.0038

Max Act: 0.9145. Min Act: -0.0001

Data Index: 20671 (C4 (Web Text))

Max Activating Token Index: 841

Text #10

Max Range: 1.0038. Min Range: -1.0038

Max Act: 0.8594. Min Act: -0.0001

Data Index: 540335 (C4 (Web Text))

Max Activating Token Index: 695

Text #11

Max Range: 1.0038. Min Range: -1.0038

Max Act: 0.7261. Min Act: -0.0001

Data Index: 1268917 (C4 (Web Text))

Max Activating Token Index: 634

Text #12

Max Range: 1.0038. Min Range: -1.0038

Max Act: 0.8518. Min Act: -0.0001

Data Index: 556911 (C4 (Web Text))

Max Activating Token Index: 445

Text #13

Max Range: 1.0038. Min Range: -1.0038

Max Act: 0.9052. Min Act: -0.0001

Data Index: 577766 (C4 (Web Text))

Max Activating Token Index: 829

Text #14

Max Range: 1.0038. Min Range: -1.0038

Max Act: 0.8359. Min Act: -0.0001

Data Index: 335805 (C4 (Web Text))

Max Activating Token Index: 124

Text #15

Max Range: 1.0038. Min Range: -1.0038

Max Act: 0.8267. Min Act: -0.0001

Data Index: 774500 (C4 (Web Text))

Max Activating Token Index: 569

Text #16

Max Range: 1.0038. Min Range: -1.0038

Max Act: 0.8546. Min Act: -0.0001

Data Index: 117711 (C4 (Web Text))

Max Activating Token Index: 128

Text #17

Max Range: 1.0038. Min Range: -1.0038

Max Act: 0.8970. Min Act: -0.0001

Data Index: 366413 (C4 (Web Text))

Max Activating Token Index: 672

Text #18

Max Range: 1.0038. Min Range: -1.0038

Max Act: 0.8002. Min Act: -0.0001

Data Index: 860774 (C4 (Web Text))

Max Activating Token Index: 805

Text #19

Max Range: 1.0038. Min Range: -1.0038

Max Act: 0.8023. Min Act: -0.0001

Data Index: 703993 (C4 (Web Text))

Max Activating Token Index: 964