Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3505 in Layer 0

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.0822. Min Range: -0.0822

Max Act: 0.0822. Min Act: -0.0002

Data Index: 469224 (C4 (Web Text))

Max Activating Token Index: 193

Text #1

Max Range: 0.0822. Min Range: -0.0822

Max Act: 0.0718. Min Act: -0.0002

Data Index: 1061527 (C4 (Web Text))

Max Activating Token Index: 625

Text #2

Max Range: 0.0822. Min Range: -0.0822

Max Act: 0.0693. Min Act: -0.0002

Data Index: 1126082 (C4 (Web Text))

Max Activating Token Index: 268

Text #3

Max Range: 0.0822. Min Range: -0.0822

Max Act: 0.0636. Min Act: -0.0002

Data Index: 128103 (C4 (Web Text))

Max Activating Token Index: 254

Text #4

Max Range: 0.0822. Min Range: -0.0822

Max Act: 0.0626. Min Act: -0.0002

Data Index: 428821 (C4 (Web Text))

Max Activating Token Index: 520

Text #5

Max Range: 0.0822. Min Range: -0.0822

Max Act: 0.0612. Min Act: -0.0002

Data Index: 121002 (C4 (Web Text))

Max Activating Token Index: 247

Text #6

Max Range: 0.0822. Min Range: -0.0822

Max Act: 0.0601. Min Act: -0.0002

Data Index: 1142618 (C4 (Web Text))

Max Activating Token Index: 185

Text #7

Max Range: 0.0822. Min Range: -0.0822

Max Act: 0.0590. Min Act: -0.0002

Data Index: 1082085 (C4 (Web Text))

Max Activating Token Index: 853

Text #8

Max Range: 0.0822. Min Range: -0.0822

Max Act: 0.0581. Min Act: -0.0002

Data Index: 480236 (C4 (Web Text))

Max Activating Token Index: 476

Text #9

Max Range: 0.0822. Min Range: -0.0822

Max Act: 0.0566. Min Act: -0.0002

Data Index: 675738 (C4 (Web Text))

Max Activating Token Index: 571

Text #10

Max Range: 0.0822. Min Range: -0.0822

Max Act: 0.0560. Min Act: -0.0002

Data Index: 277907 (C4 (Web Text))

Max Activating Token Index: 218

Text #11

Max Range: 0.0822. Min Range: -0.0822

Max Act: 0.0559. Min Act: -0.0002

Data Index: 426077 (C4 (Web Text))

Max Activating Token Index: 232

Text #12

Max Range: 0.0822. Min Range: -0.0822

Max Act: 0.0539. Min Act: -0.0002

Data Index: 306805 (C4 (Web Text))

Max Activating Token Index: 269

Text #13

Max Range: 0.0822. Min Range: -0.0822

Max Act: 0.0523. Min Act: -0.0002

Data Index: 866883 (C4 (Web Text))

Max Activating Token Index: 112

Text #14

Max Range: 0.0822. Min Range: -0.0822

Max Act: 0.0541. Min Act: -0.0002

Data Index: 1164332 (C4 (Web Text))

Max Activating Token Index: 313

Text #15

Max Range: 0.0822. Min Range: -0.0822

Max Act: 0.0539. Min Act: -0.0002

Data Index: 762624 (C4 (Web Text))

Max Activating Token Index: 267

Text #16

Max Range: 0.0822. Min Range: -0.0822

Max Act: 0.0527. Min Act: -0.0002

Data Index: 46943 (C4 (Web Text))

Max Activating Token Index: 487

Text #17

Max Range: 0.0822. Min Range: -0.0822

Max Act: 0.0537. Min Act: -0.0002

Data Index: 48484 (C4 (Web Text))

Max Activating Token Index: 420

Text #18

Max Range: 0.0822. Min Range: -0.0822

Max Act: 0.0518. Min Act: -0.0002

Data Index: 1067746 (C4 (Web Text))

Max Activating Token Index: 937

Text #19

Max Range: 0.0822. Min Range: -0.0822

Max Act: 0.0518. Min Act: -0.0002

Data Index: 892795 (C4 (Web Text))

Max Activating Token Index: 160