Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 598 in Layer 3

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.7970. Min Range: -0.7970

Max Act: 0.7970. Min Act: -0.0001

Data Index: 312703 (C4 (Web Text))

Max Activating Token Index: 127

Text #1

Max Range: 0.7970. Min Range: -0.7970

Max Act: 0.7684. Min Act: -0.0001

Data Index: 895763 (C4 (Web Text))

Max Activating Token Index: 280

Text #2

Max Range: 0.7970. Min Range: -0.7970

Max Act: 0.6839. Min Act: -0.0001

Data Index: 799492 (C4 (Web Text))

Max Activating Token Index: 154

Text #3

Max Range: 0.7970. Min Range: -0.7970

Max Act: 0.6768. Min Act: -0.0001

Data Index: 1302719 (C4 (Web Text))

Max Activating Token Index: 612

Text #4

Max Range: 0.7970. Min Range: -0.7970

Max Act: 0.6724. Min Act: -0.0001

Data Index: 300394 (C4 (Web Text))

Max Activating Token Index: 958

Text #5

Max Range: 0.7970. Min Range: -0.7970

Max Act: 0.7307. Min Act: -0.0001

Data Index: 772450 (C4 (Web Text))

Max Activating Token Index: 220

Text #6

Max Range: 0.7970. Min Range: -0.7970

Max Act: 0.6587. Min Act: -0.0001

Data Index: 1191833 (C4 (Web Text))

Max Activating Token Index: 106

Text #7

Max Range: 0.7970. Min Range: -0.7970

Max Act: 0.7410. Min Act: -0.0001

Data Index: 1302317 (C4 (Web Text))

Max Activating Token Index: 747

Text #8

Max Range: 0.7970. Min Range: -0.7970

Max Act: 0.6552. Min Act: -0.0001

Data Index: 807373 (C4 (Web Text))

Max Activating Token Index: 317

Text #9

Max Range: 0.7970. Min Range: -0.7970

Max Act: 0.7418. Min Act: -0.0001

Data Index: 422485 (C4 (Web Text))

Max Activating Token Index: 83

Text #10

Max Range: 0.7970. Min Range: -0.7970

Max Act: 0.6653. Min Act: -0.0001

Data Index: 64150 (C4 (Web Text))

Max Activating Token Index: 366

Text #11

Max Range: 0.7970. Min Range: -0.7970

Max Act: 0.7404. Min Act: -0.0001

Data Index: 1259421 (C4 (Web Text))

Max Activating Token Index: 223

Text #12

Max Range: 0.7970. Min Range: -0.7970

Max Act: 0.6722. Min Act: -0.0001

Data Index: 1241092 (C4 (Web Text))

Max Activating Token Index: 401

Text #13

Max Range: 0.7970. Min Range: -0.7970

Max Act: 0.6818. Min Act: -0.0001

Data Index: 353905 (C4 (Web Text))

Max Activating Token Index: 190

Text #14

Max Range: 0.7970. Min Range: -0.7970

Max Act: 0.6563. Min Act: -0.0001

Data Index: 573231 (C4 (Web Text))

Max Activating Token Index: 530

Text #15

Max Range: 0.7970. Min Range: -0.7970

Max Act: 0.6513. Min Act: -0.0001

Data Index: 624134 (C4 (Web Text))

Max Activating Token Index: 780

Text #16

Max Range: 0.7970. Min Range: -0.7970

Max Act: 0.6479. Min Act: -0.0001

Data Index: 1359273 (C4 (Web Text))

Max Activating Token Index: 546

Text #17

Max Range: 0.7970. Min Range: -0.7970

Max Act: 0.5778. Min Act: -0.0001

Data Index: 1132385 (C4 (Web Text))

Max Activating Token Index: 79

Text #18

Max Range: 0.7970. Min Range: -0.7970

Max Act: 0.5963. Min Act: -0.0001

Data Index: 1343433 (C4 (Web Text))

Max Activating Token Index: 47

Text #19

Max Range: 0.7970. Min Range: -0.7970

Max Act: 0.7000. Min Act: -0.0001

Data Index: 861887 (C4 (Web Text))

Max Activating Token Index: 447