Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2960 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.1403. Min Range: -1.1403

Max Act: 1.0962. Min Act: -0.0000

Data Index: 1233031 (C4 (Web Text))

Max Activating Token Index: 621

Text #1

Max Range: 1.1403. Min Range: -1.1403

Max Act: 1.1133. Min Act: -0.0000

Data Index: 813252 (C4 (Web Text))

Max Activating Token Index: 866

Text #2

Max Range: 1.1403. Min Range: -1.1403

Max Act: 1.1403. Min Act: -0.0000

Data Index: 1212526 (C4 (Web Text))

Max Activating Token Index: 280

Text #3

Max Range: 1.1403. Min Range: -1.1403

Max Act: 1.0919. Min Act: -0.0000

Data Index: 643105 (C4 (Web Text))

Max Activating Token Index: 841

Text #4

Max Range: 1.1403. Min Range: -1.1403

Max Act: 1.0881. Min Act: -0.0000

Data Index: 1190463 (C4 (Web Text))

Max Activating Token Index: 752

Text #5

Max Range: 1.1403. Min Range: -1.1403

Max Act: 1.0585. Min Act: -0.0000

Data Index: 1123843 (C4 (Web Text))

Max Activating Token Index: 506

Text #6

Max Range: 1.1403. Min Range: -1.1403

Max Act: 1.0503. Min Act: -0.0000

Data Index: 254319 (C4 (Web Text))

Max Activating Token Index: 765

Text #7

Max Range: 1.1403. Min Range: -1.1403

Max Act: 1.0857. Min Act: -0.0000

Data Index: 230398 (C4 (Web Text))

Max Activating Token Index: 710

Text #8

Max Range: 1.1403. Min Range: -1.1403

Max Act: 1.0831. Min Act: -0.0000

Data Index: 1066003 (C4 (Web Text))

Max Activating Token Index: 524

Text #9

Max Range: 1.1403. Min Range: -1.1403

Max Act: 1.0443. Min Act: -0.0000

Data Index: 107857 (C4 (Web Text))

Max Activating Token Index: 911

Text #10

Max Range: 1.1403. Min Range: -1.1403

Max Act: 1.0165. Min Act: -0.0000

Data Index: 490769 (C4 (Web Text))

Max Activating Token Index: 651

Text #11

Max Range: 1.1403. Min Range: -1.1403

Max Act: 1.0404. Min Act: -0.0000

Data Index: 595598 (C4 (Web Text))

Max Activating Token Index: 865

Text #12

Max Range: 1.1403. Min Range: -1.1403

Max Act: 1.0050. Min Act: -0.0000

Data Index: 931940 (C4 (Web Text))

Max Activating Token Index: 283

Text #13

Max Range: 1.1403. Min Range: -1.1403

Max Act: 1.0008. Min Act: -0.0000

Data Index: 974283 (C4 (Web Text))

Max Activating Token Index: 498

Text #14

Max Range: 1.1403. Min Range: -1.1403

Max Act: 1.0582. Min Act: -0.0000

Data Index: 1046903 (C4 (Web Text))

Max Activating Token Index: 493

Text #15

Max Range: 1.1403. Min Range: -1.1403

Max Act: 1.0242. Min Act: -0.0000

Data Index: 190474 (C4 (Web Text))

Max Activating Token Index: 606

Text #16

Max Range: 1.1403. Min Range: -1.1403

Max Act: 0.9935. Min Act: -0.0000

Data Index: 32515 (C4 (Web Text))

Max Activating Token Index: 743

Text #17

Max Range: 1.1403. Min Range: -1.1403

Max Act: 0.9885. Min Act: -0.0000

Data Index: 1133677 (C4 (Web Text))

Max Activating Token Index: 801

Text #18

Max Range: 1.1403. Min Range: -1.1403

Max Act: 1.0528. Min Act: -0.0000

Data Index: 761867 (C4 (Web Text))

Max Activating Token Index: 460

Text #19

Max Range: 1.1403. Min Range: -1.1403

Max Act: 0.9974. Min Act: -0.0000

Data Index: 860615 (C4 (Web Text))

Max Activating Token Index: 788