Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4585 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 2.0664. Min Range: -2.0664

Max Act: 2.0664. Min Act: -0.0000

Data Index: 22256 (C4 (Web Text))

Max Activating Token Index: 948

Text #1

Max Range: 2.0664. Min Range: -2.0664

Max Act: 1.8296. Min Act: -0.0000

Data Index: 301388 (C4 (Web Text))

Max Activating Token Index: 480

Text #2

Max Range: 2.0664. Min Range: -2.0664

Max Act: 1.9090. Min Act: -0.0000

Data Index: 468680 (C4 (Web Text))

Max Activating Token Index: 831

Text #3

Max Range: 2.0664. Min Range: -2.0664

Max Act: 1.3773. Min Act: -0.0000

Data Index: 123536 (C4 (Web Text))

Max Activating Token Index: 772

Text #4

Max Range: 2.0664. Min Range: -2.0664

Max Act: 1.4396. Min Act: -0.0000

Data Index: 1323656 (C4 (Web Text))

Max Activating Token Index: 868

Text #5

Max Range: 2.0664. Min Range: -2.0664

Max Act: 1.4114. Min Act: 0.0000

Data Index: 944793 (C4 (Web Text))

Max Activating Token Index: 563

Text #6

Max Range: 2.0664. Min Range: -2.0664

Max Act: 1.4921. Min Act: 0.0000

Data Index: 1333069 (C4 (Web Text))

Max Activating Token Index: 827

Text #7

Max Range: 2.0664. Min Range: -2.0664

Max Act: 1.2902. Min Act: -0.0000

Data Index: 512483 (C4 (Web Text))

Max Activating Token Index: 582

Text #8

Max Range: 2.0664. Min Range: -2.0664

Max Act: 1.3544. Min Act: -0.0000

Data Index: 828538 (C4 (Web Text))

Max Activating Token Index: 722

Text #9

Max Range: 2.0664. Min Range: -2.0664

Max Act: 1.2440. Min Act: -0.0000

Data Index: 1164600 (C4 (Web Text))

Max Activating Token Index: 452

Text #10

Max Range: 2.0664. Min Range: -2.0664

Max Act: 1.3932. Min Act: -0.0000

Data Index: 136464 (C4 (Web Text))

Max Activating Token Index: 846

Text #11

Max Range: 2.0664. Min Range: -2.0664

Max Act: 1.2370. Min Act: -0.0000

Data Index: 404159 (C4 (Web Text))

Max Activating Token Index: 375

Text #12

Max Range: 2.0664. Min Range: -2.0664

Max Act: 1.1903. Min Act: -0.0000

Data Index: 1115476 (C4 (Web Text))

Max Activating Token Index: 449

Text #13

Max Range: 2.0664. Min Range: -2.0664

Max Act: 1.2075. Min Act: -0.0000

Data Index: 522972 (C4 (Web Text))

Max Activating Token Index: 372

Text #14

Max Range: 2.0664. Min Range: -2.0664

Max Act: 1.2007. Min Act: -0.0000

Data Index: 93970 (C4 (Web Text))

Max Activating Token Index: 688

Text #15

Max Range: 2.0664. Min Range: -2.0664

Max Act: 1.1959. Min Act: -0.0000

Data Index: 658405 (C4 (Web Text))

Max Activating Token Index: 249

Text #16

Max Range: 2.0664. Min Range: -2.0664

Max Act: 1.1898. Min Act: -0.0000

Data Index: 457107 (C4 (Web Text))

Max Activating Token Index: 131

Text #17

Max Range: 2.0664. Min Range: -2.0664

Max Act: 1.1960. Min Act: -0.0000

Data Index: 1230330 (C4 (Web Text))

Max Activating Token Index: 71

Text #18

Max Range: 2.0664. Min Range: -2.0664

Max Act: 1.1890. Min Act: -0.0000

Data Index: 474891 (C4 (Web Text))

Max Activating Token Index: 840

Text #19

Max Range: 2.0664. Min Range: -2.0664

Max Act: 1.1794. Min Act: -0.0000

Data Index: 199397 (C4 (Web Text))

Max Activating Token Index: 64