Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4972 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.5738. Min Range: -0.5738

Max Act: 0.5738. Min Act: -0.0000

Data Index: 1287973 (C4 (Web Text))

Max Activating Token Index: 22

Text #1

Max Range: 0.5738. Min Range: -0.5738

Max Act: 0.3917. Min Act: 0.0000

Data Index: 1023884 (C4 (Web Text))

Max Activating Token Index: 339

Text #2

Max Range: 0.5738. Min Range: -0.5738

Max Act: 0.4043. Min Act: -0.0000

Data Index: 458308 (C4 (Web Text))

Max Activating Token Index: 536

Text #3

Max Range: 0.5738. Min Range: -0.5738

Max Act: 0.4023. Min Act: -0.0000

Data Index: 719943 (C4 (Web Text))

Max Activating Token Index: 198

Text #4

Max Range: 0.5738. Min Range: -0.5738

Max Act: 0.3900. Min Act: -0.0000

Data Index: 726851 (C4 (Web Text))

Max Activating Token Index: 408

Text #5

Max Range: 0.5738. Min Range: -0.5738

Max Act: 0.4019. Min Act: -0.0000

Data Index: 624551 (C4 (Web Text))

Max Activating Token Index: 736

Text #6

Max Range: 0.5738. Min Range: -0.5738

Max Act: 0.3708. Min Act: -0.0000

Data Index: 239845 (C4 (Web Text))

Max Activating Token Index: 291

Text #7

Max Range: 0.5738. Min Range: -0.5738

Max Act: 0.3655. Min Act: -0.0000

Data Index: 1061351 (C4 (Web Text))

Max Activating Token Index: 135

Text #8

Max Range: 0.5738. Min Range: -0.5738

Max Act: 0.3651. Min Act: -0.0000

Data Index: 675132 (C4 (Web Text))

Max Activating Token Index: 912

Text #9

Max Range: 0.5738. Min Range: -0.5738

Max Act: 0.3528. Min Act: -0.0000

Data Index: 574572 (C4 (Web Text))

Max Activating Token Index: 43

Text #10

Max Range: 0.5738. Min Range: -0.5738

Max Act: 0.3686. Min Act: -0.0000

Data Index: 134071 (C4 (Web Text))

Max Activating Token Index: 748

Text #11

Max Range: 0.5738. Min Range: -0.5738

Max Act: 0.3572. Min Act: -0.0000

Data Index: 3146 (C4 (Web Text))

Max Activating Token Index: 39

Text #12

Max Range: 0.5738. Min Range: -0.5738

Max Act: 0.3520. Min Act: -0.0000

Data Index: 623852 (C4 (Web Text))

Max Activating Token Index: 589

Text #13

Max Range: 0.5738. Min Range: -0.5738

Max Act: 0.3424. Min Act: -0.0000

Data Index: 917752 (C4 (Web Text))

Max Activating Token Index: 52

Text #14

Max Range: 0.5738. Min Range: -0.5738

Max Act: 0.3406. Min Act: -0.0000

Data Index: 586366 (C4 (Web Text))

Max Activating Token Index: 382

Text #15

Max Range: 0.5738. Min Range: -0.5738

Max Act: 0.3473. Min Act: -0.0000

Data Index: 853156 (C4 (Web Text))

Max Activating Token Index: 989

Text #16

Max Range: 0.5738. Min Range: -0.5738

Max Act: 0.3326. Min Act: -0.0000

Data Index: 899044 (C4 (Web Text))

Max Activating Token Index: 534

Text #17

Max Range: 0.5738. Min Range: -0.5738

Max Act: 0.3401. Min Act: -0.0000

Data Index: 912385 (C4 (Web Text))

Max Activating Token Index: 751

Text #18

Max Range: 0.5738. Min Range: -0.5738

Max Act: 0.3350. Min Act: -0.0000

Data Index: 1138444 (C4 (Web Text))

Max Activating Token Index: 878

Text #19

Max Range: 0.5738. Min Range: -0.5738

Max Act: 0.3221. Min Act: -0.0000

Data Index: 1034024 (C4 (Web Text))

Max Activating Token Index: 302