Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1594 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.2000. Min Range: -0.2000

Max Act: 0.2000. Min Act: -0.0000

Data Index: 846724 (C4 (Web Text))

Max Activating Token Index: 403

Text #1

Max Range: 0.2000. Min Range: -0.2000

Max Act: 0.1840. Min Act: -0.0000

Data Index: 638869 (C4 (Web Text))

Max Activating Token Index: 296

Text #2

Max Range: 0.2000. Min Range: -0.2000

Max Act: 0.1883. Min Act: -0.0000

Data Index: 22773 (C4 (Web Text))

Max Activating Token Index: 761

Text #3

Max Range: 0.2000. Min Range: -0.2000

Max Act: 0.1691. Min Act: -0.0000

Data Index: 1047375 (C4 (Web Text))

Max Activating Token Index: 255

Text #4

Max Range: 0.2000. Min Range: -0.2000

Max Act: 0.1736. Min Act: -0.0000

Data Index: 239395 (C4 (Web Text))

Max Activating Token Index: 145

Text #5

Max Range: 0.2000. Min Range: -0.2000

Max Act: 0.1718. Min Act: -0.0000

Data Index: 736344 (C4 (Web Text))

Max Activating Token Index: 572

Text #6

Max Range: 0.2000. Min Range: -0.2000

Max Act: 0.1719. Min Act: -0.0000

Data Index: 815044 (C4 (Web Text))

Max Activating Token Index: 632

Text #7

Max Range: 0.2000. Min Range: -0.2000

Max Act: 0.1699. Min Act: -0.0000

Data Index: 931526 (C4 (Web Text))

Max Activating Token Index: 549

Text #8

Max Range: 0.2000. Min Range: -0.2000

Max Act: 0.1742. Min Act: 0.0000

Data Index: 194733 (C4 (Web Text))

Max Activating Token Index: 681

Text #9

Max Range: 0.2000. Min Range: -0.2000

Max Act: 0.1639. Min Act: -0.0000

Data Index: 492307 (C4 (Web Text))

Max Activating Token Index: 331

Text #10

Max Range: 0.2000. Min Range: -0.2000

Max Act: 0.1615. Min Act: -0.0000

Data Index: 234934 (C4 (Web Text))

Max Activating Token Index: 611

Text #11

Max Range: 0.2000. Min Range: -0.2000

Max Act: 0.1593. Min Act: -0.0000

Data Index: 1042905 (C4 (Web Text))

Max Activating Token Index: 486

Text #12

Max Range: 0.2000. Min Range: -0.2000

Max Act: 0.1648. Min Act: -0.0000

Data Index: 300134 (C4 (Web Text))

Max Activating Token Index: 697

Text #13

Max Range: 0.2000. Min Range: -0.2000

Max Act: 0.1574. Min Act: -0.0000

Data Index: 751920 (C4 (Web Text))

Max Activating Token Index: 1015

Text #14

Max Range: 0.2000. Min Range: -0.2000

Max Act: 0.1600. Min Act: -0.0000

Data Index: 1051564 (C4 (Web Text))

Max Activating Token Index: 499

Text #15

Max Range: 0.2000. Min Range: -0.2000

Max Act: 0.1550. Min Act: -0.0000

Data Index: 13620 (C4 (Web Text))

Max Activating Token Index: 191

Text #16

Max Range: 0.2000. Min Range: -0.2000

Max Act: 0.1482. Min Act: 0.0000

Data Index: 966513 (C4 (Web Text))

Max Activating Token Index: 1009

Text #17

Max Range: 0.2000. Min Range: -0.2000

Max Act: 0.1533. Min Act: -0.0000

Data Index: 920492 (C4 (Web Text))

Max Activating Token Index: 440

Text #18

Max Range: 0.2000. Min Range: -0.2000

Max Act: 0.1484. Min Act: -0.0000

Data Index: 966246 (C4 (Web Text))

Max Activating Token Index: 128

Text #19

Max Range: 0.2000. Min Range: -0.2000

Max Act: 0.1523. Min Act: -0.0000

Data Index: 1092881 (C4 (Web Text))

Max Activating Token Index: 354