Model: SoLU Model: 8 Layers, 4096 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1844 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-8l')

Text #0

Max Range: 0.3729. Min Range: -0.3729

Max Act: 0.3729. Min Act: -0.0000

Data Index: 90908 (C4 (Web Text))

Max Activating Token Index: 639

Text #1

Max Range: 0.3729. Min Range: -0.3729

Max Act: 0.3452. Min Act: -0.0001

Data Index: 30781 (C4 (Web Text))

Max Activating Token Index: 841

Text #2

Max Range: 0.3729. Min Range: -0.3729

Max Act: 0.3534. Min Act: -0.0000

Data Index: 1132949 (C4 (Web Text))

Max Activating Token Index: 1001

Text #3

Max Range: 0.3729. Min Range: -0.3729

Max Act: 0.3390. Min Act: -0.0001

Data Index: 96558 (C4 (Web Text))

Max Activating Token Index: 627

Text #4

Max Range: 0.3729. Min Range: -0.3729

Max Act: 0.3156. Min Act: -0.0000

Data Index: 483268 (C4 (Web Text))

Max Activating Token Index: 294

Text #5

Max Range: 0.3729. Min Range: -0.3729

Max Act: 0.3167. Min Act: -0.0001

Data Index: 377621 (C4 (Web Text))

Max Activating Token Index: 63

Text #6

Max Range: 0.3729. Min Range: -0.3729

Max Act: 0.2943. Min Act: -0.0000

Data Index: 69521 (C4 (Web Text))

Max Activating Token Index: 718

Text #7

Max Range: 0.3729. Min Range: -0.3729

Max Act: 0.3051. Min Act: -0.0000

Data Index: 874760 (C4 (Web Text))

Max Activating Token Index: 729

Text #8

Max Range: 0.3729. Min Range: -0.3729

Max Act: 0.3108. Min Act: -0.0001

Data Index: 1209010 (C4 (Web Text))

Max Activating Token Index: 299

Text #9

Max Range: 0.3729. Min Range: -0.3729

Max Act: 0.3081. Min Act: -0.0001

Data Index: 140533 (C4 (Web Text))

Max Activating Token Index: 679

Text #10

Max Range: 0.3729. Min Range: -0.3729

Max Act: 0.2834. Min Act: -0.0001

Data Index: 693741 (C4 (Web Text))

Max Activating Token Index: 936

Text #11

Max Range: 0.3729. Min Range: -0.3729

Max Act: 0.2915. Min Act: -0.0001

Data Index: 1196803 (C4 (Web Text))

Max Activating Token Index: 630

Text #12

Max Range: 0.3729. Min Range: -0.3729

Max Act: 0.2769. Min Act: -0.0001

Data Index: 8121 (C4 (Web Text))

Max Activating Token Index: 756

Text #13

Max Range: 0.3729. Min Range: -0.3729

Max Act: 0.2857. Min Act: -0.0000

Data Index: 1290173 (C4 (Web Text))

Max Activating Token Index: 563

Text #14

Max Range: 0.3729. Min Range: -0.3729

Max Act: 0.2714. Min Act: -0.0000

Data Index: 808837 (C4 (Web Text))

Max Activating Token Index: 374

Text #15

Max Range: 0.3729. Min Range: -0.3729

Max Act: 0.2617. Min Act: -0.0001

Data Index: 186163 (C4 (Web Text))

Max Activating Token Index: 308

Text #16

Max Range: 0.3729. Min Range: -0.3729

Max Act: 0.2576. Min Act: -0.0000

Data Index: 1037795 (C4 (Web Text))

Max Activating Token Index: 33

Text #17

Max Range: 0.3729. Min Range: -0.3729

Max Act: 0.2756. Min Act: -0.0001

Data Index: 115545 (C4 (Web Text))

Max Activating Token Index: 277

Text #18

Max Range: 0.3729. Min Range: -0.3729

Max Act: 0.2637. Min Act: -0.0001

Data Index: 3850 (C4 (Web Text))

Max Activating Token Index: 638

Text #19

Max Range: 0.3729. Min Range: -0.3729

Max Act: 0.2640. Min Act: -0.0000

Data Index: 640346 (C4 (Web Text))

Max Activating Token Index: 986