Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4676 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.2567. Min Range: -0.2567

Max Act: 0.2567. Min Act: -0.0000

Data Index: 866695 (C4 (Web Text))

Max Activating Token Index: 613

Text #1

Max Range: 0.2567. Min Range: -0.2567

Max Act: 0.2254. Min Act: -0.0000

Data Index: 1199204 (C4 (Web Text))

Max Activating Token Index: 235

Text #2

Max Range: 0.2567. Min Range: -0.2567

Max Act: 0.2466. Min Act: -0.0000

Data Index: 789152 (C4 (Web Text))

Max Activating Token Index: 256

Text #3

Max Range: 0.2567. Min Range: -0.2567

Max Act: 0.2238. Min Act: -0.0000

Data Index: 887775 (C4 (Web Text))

Max Activating Token Index: 949

Text #4

Max Range: 0.2567. Min Range: -0.2567

Max Act: 0.2224. Min Act: -0.0000

Data Index: 1140316 (C4 (Web Text))

Max Activating Token Index: 534

Text #5

Max Range: 0.2567. Min Range: -0.2567

Max Act: 0.2031. Min Act: -0.0000

Data Index: 18759 (C4 (Web Text))

Max Activating Token Index: 389

Text #6

Max Range: 0.2567. Min Range: -0.2567

Max Act: 0.1954. Min Act: -0.0000

Data Index: 1345189 (C4 (Web Text))

Max Activating Token Index: 546

Text #7

Max Range: 0.2567. Min Range: -0.2567

Max Act: 0.1732. Min Act: -0.0000

Data Index: 857634 (C4 (Web Text))

Max Activating Token Index: 43

Text #8

Max Range: 0.2567. Min Range: -0.2567

Max Act: 0.1984. Min Act: -0.0000

Data Index: 1072324 (C4 (Web Text))

Max Activating Token Index: 97

Text #9

Max Range: 0.2567. Min Range: -0.2567

Max Act: 0.1784. Min Act: -0.0000

Data Index: 1341895 (C4 (Web Text))

Max Activating Token Index: 299

Text #10

Max Range: 0.2567. Min Range: -0.2567

Max Act: 0.1834. Min Act: -0.0000

Data Index: 287350 (C4 (Web Text))

Max Activating Token Index: 595

Text #11

Max Range: 0.2567. Min Range: -0.2567

Max Act: 0.1738. Min Act: -0.0000

Data Index: 481145 (C4 (Web Text))

Max Activating Token Index: 1023

Text #12

Max Range: 0.2567. Min Range: -0.2567

Max Act: 0.1726. Min Act: -0.0000

Data Index: 47453 (C4 (Web Text))

Max Activating Token Index: 918

Text #13

Max Range: 0.2567. Min Range: -0.2567

Max Act: 0.1549. Min Act: -0.0000

Data Index: 461328 (C4 (Web Text))

Max Activating Token Index: 772

Text #14

Max Range: 0.2567. Min Range: -0.2567

Max Act: 0.1709. Min Act: -0.0000

Data Index: 171932 (C4 (Web Text))

Max Activating Token Index: 67

Text #15

Max Range: 0.2567. Min Range: -0.2567

Max Act: 0.1631. Min Act: -0.0000

Data Index: 1303930 (C4 (Web Text))

Max Activating Token Index: 536

Text #16

Max Range: 0.2567. Min Range: -0.2567

Max Act: 0.1760. Min Act: -0.0000

Data Index: 1198384 (C4 (Web Text))

Max Activating Token Index: 270

Text #17

Max Range: 0.2567. Min Range: -0.2567

Max Act: 0.1577. Min Act: -0.0000

Data Index: 1011277 (C4 (Web Text))

Max Activating Token Index: 937

Text #18

Max Range: 0.2567. Min Range: -0.2567

Max Act: 0.1623. Min Act: -0.0000

Data Index: 793030 (C4 (Web Text))

Max Activating Token Index: 380

Text #19

Max Range: 0.2567. Min Range: -0.2567

Max Act: 0.1556. Min Act: -0.0000

Data Index: 27394 (C4 (Web Text))

Max Activating Token Index: 173