Model: SoLU Model: 8 Layers, 4096 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1369 in Layer 3

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-8l')

Text #0

Max Range: 1.2480. Min Range: -1.2480

Max Act: 1.2480. Min Act: -0.0001

Data Index: 743229 (C4 (Web Text))

Max Activating Token Index: 600

Text #1

Max Range: 1.2480. Min Range: -1.2480

Max Act: 1.1634. Min Act: -0.0001

Data Index: 411167 (C4 (Web Text))

Max Activating Token Index: 760

Text #2

Max Range: 1.2480. Min Range: -1.2480

Max Act: 1.1357. Min Act: -0.0001

Data Index: 922490 (C4 (Web Text))

Max Activating Token Index: 454

Text #3

Max Range: 1.2480. Min Range: -1.2480

Max Act: 1.1513. Min Act: -0.0001

Data Index: 451614 (C4 (Web Text))

Max Activating Token Index: 876

Text #4

Max Range: 1.2480. Min Range: -1.2480

Max Act: 1.0569. Min Act: -0.0001

Data Index: 1311894 (C4 (Web Text))

Max Activating Token Index: 90

Text #5

Max Range: 1.2480. Min Range: -1.2480

Max Act: 0.9726. Min Act: -0.0001

Data Index: 554300 (C4 (Web Text))

Max Activating Token Index: 621

Text #6

Max Range: 1.2480. Min Range: -1.2480

Max Act: 1.1290. Min Act: -0.0001

Data Index: 780788 (C4 (Web Text))

Max Activating Token Index: 933

Text #7

Max Range: 1.2480. Min Range: -1.2480

Max Act: 1.0937. Min Act: -0.0001

Data Index: 955883 (C4 (Web Text))

Max Activating Token Index: 878

Text #8

Max Range: 1.2480. Min Range: -1.2480

Max Act: 1.0199. Min Act: -0.0001

Data Index: 979895 (C4 (Web Text))

Max Activating Token Index: 851

Text #9

Max Range: 1.2480. Min Range: -1.2480

Max Act: 0.9125. Min Act: -0.0001

Data Index: 372747 (C4 (Web Text))

Max Activating Token Index: 369

Text #10

Max Range: 1.2480. Min Range: -1.2480

Max Act: 0.9362. Min Act: -0.0001

Data Index: 227006 (C4 (Web Text))

Max Activating Token Index: 334

Text #11

Max Range: 1.2480. Min Range: -1.2480

Max Act: 0.9651. Min Act: -0.0001

Data Index: 630551 (C4 (Web Text))

Max Activating Token Index: 839

Text #12

Max Range: 1.2480. Min Range: -1.2480

Max Act: 0.9136. Min Act: -0.0001

Data Index: 81649 (C4 (Web Text))

Max Activating Token Index: 440

Text #13

Max Range: 1.2480. Min Range: -1.2480

Max Act: 0.8948. Min Act: -0.0001

Data Index: 848529 (C4 (Web Text))

Max Activating Token Index: 889

Text #14

Max Range: 1.2480. Min Range: -1.2480

Max Act: 0.8847. Min Act: -0.0001

Data Index: 844932 (C4 (Web Text))

Max Activating Token Index: 362

Text #15

Max Range: 1.2480. Min Range: -1.2480

Max Act: 0.9119. Min Act: -0.0001

Data Index: 66545 (C4 (Web Text))

Max Activating Token Index: 119

Text #16

Max Range: 1.2480. Min Range: -1.2480

Max Act: 0.9004. Min Act: -0.0001

Data Index: 660510 (C4 (Web Text))

Max Activating Token Index: 455

Text #17

Max Range: 1.2480. Min Range: -1.2480

Max Act: 0.9064. Min Act: -0.0001

Data Index: 1012194 (C4 (Web Text))

Max Activating Token Index: 410

Text #18

Max Range: 1.2480. Min Range: -1.2480

Max Act: 0.8427. Min Act: -0.0001

Data Index: 258364 (C4 (Web Text))

Max Activating Token Index: 977

Text #19

Max Range: 1.2480. Min Range: -1.2480

Max Act: 0.8519. Min Act: -0.0001

Data Index: 557066 (C4 (Web Text))

Max Activating Token Index: 415