Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1999 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 3.2911. Min Range: -3.2911

Max Act: 3.2911. Min Act: -0.0000

Data Index: 296663 (C4 (Web Text))

Max Activating Token Index: 295

Text #1

Max Range: 3.2911. Min Range: -3.2911

Max Act: 2.8621. Min Act: -0.0000

Data Index: 693481 (C4 (Web Text))

Max Activating Token Index: 457

Text #2

Max Range: 3.2911. Min Range: -3.2911

Max Act: 2.9776. Min Act: -0.0000

Data Index: 1110021 (C4 (Web Text))

Max Activating Token Index: 409

Text #3

Max Range: 3.2911. Min Range: -3.2911

Max Act: 2.5503. Min Act: -0.0000

Data Index: 825585 (C4 (Web Text))

Max Activating Token Index: 1008

Text #4

Max Range: 3.2911. Min Range: -3.2911

Max Act: 2.6281. Min Act: -0.0000

Data Index: 354955 (C4 (Web Text))

Max Activating Token Index: 936

Text #5

Max Range: 3.2911. Min Range: -3.2911

Max Act: 2.5612. Min Act: -0.0000

Data Index: 554812 (C4 (Web Text))

Max Activating Token Index: 641

Text #6

Max Range: 3.2911. Min Range: -3.2911

Max Act: 2.3226. Min Act: -0.0000

Data Index: 351470 (C4 (Web Text))

Max Activating Token Index: 719

Text #7

Max Range: 3.2911. Min Range: -3.2911

Max Act: 2.1769. Min Act: -0.0000

Data Index: 347886 (C4 (Web Text))

Max Activating Token Index: 722

Text #8

Max Range: 3.2911. Min Range: -3.2911

Max Act: 2.1456. Min Act: -0.0000

Data Index: 1035115 (C4 (Web Text))

Max Activating Token Index: 605

Text #9

Max Range: 3.2911. Min Range: -3.2911

Max Act: 2.3276. Min Act: -0.0000

Data Index: 865625 (C4 (Web Text))

Max Activating Token Index: 706

Text #10

Max Range: 3.2911. Min Range: -3.2911

Max Act: 2.1668. Min Act: -0.0000

Data Index: 878935 (C4 (Web Text))

Max Activating Token Index: 26

Text #11

Max Range: 3.2911. Min Range: -3.2911

Max Act: 2.0393. Min Act: -0.0000

Data Index: 713691 (C4 (Web Text))

Max Activating Token Index: 551

Text #12

Max Range: 3.2911. Min Range: -3.2911

Max Act: 2.0211. Min Act: -0.0000

Data Index: 249566 (C4 (Web Text))

Max Activating Token Index: 286

Text #13

Max Range: 3.2911. Min Range: -3.2911

Max Act: 1.9989. Min Act: -0.0000

Data Index: 1347878 (C4 (Web Text))

Max Activating Token Index: 500

Text #14

Max Range: 3.2911. Min Range: -3.2911

Max Act: 2.0862. Min Act: -0.0000

Data Index: 1293272 (C4 (Web Text))

Max Activating Token Index: 62

Text #15

Max Range: 3.2911. Min Range: -3.2911

Max Act: 2.0603. Min Act: -0.0000

Data Index: 1005737 (C4 (Web Text))

Max Activating Token Index: 898

Text #16

Max Range: 3.2911. Min Range: -3.2911

Max Act: 1.9916. Min Act: -0.0000

Data Index: 373762 (C4 (Web Text))

Max Activating Token Index: 568

Text #17

Max Range: 3.2911. Min Range: -3.2911

Max Act: 1.7295. Min Act: -0.0000

Data Index: 467873 (C4 (Web Text))

Max Activating Token Index: 995

Text #18

Max Range: 3.2911. Min Range: -3.2911

Max Act: 1.9092. Min Act: -0.0000

Data Index: 1357045 (C4 (Web Text))

Max Activating Token Index: 320

Text #19

Max Range: 3.2911. Min Range: -3.2911

Max Act: 1.9939. Min Act: -0.0000

Data Index: 609942 (C4 (Web Text))

Max Activating Token Index: 497