Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 762 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 2.0802. Min Range: -2.0802

Max Act: 2.0802. Min Act: -0.0000

Data Index: 998797 (C4 (Web Text))

Max Activating Token Index: 484

Text #1

Max Range: 2.0802. Min Range: -2.0802

Max Act: 1.8001. Min Act: -0.0000

Data Index: 573510 (C4 (Web Text))

Max Activating Token Index: 305

Text #2

Max Range: 2.0802. Min Range: -2.0802

Max Act: 1.6702. Min Act: -0.0000

Data Index: 387290 (C4 (Web Text))

Max Activating Token Index: 575

Text #3

Max Range: 2.0802. Min Range: -2.0802

Max Act: 1.7074. Min Act: -0.0000

Data Index: 606620 (C4 (Web Text))

Max Activating Token Index: 322

Text #4

Max Range: 2.0802. Min Range: -2.0802

Max Act: 1.7934. Min Act: -0.0000

Data Index: 1033841 (C4 (Web Text))

Max Activating Token Index: 120

Text #5

Max Range: 2.0802. Min Range: -2.0802

Max Act: 1.6546. Min Act: -0.0000

Data Index: 508918 (C4 (Web Text))

Max Activating Token Index: 713

Text #6

Max Range: 2.0802. Min Range: -2.0802

Max Act: 1.6169. Min Act: -0.0000

Data Index: 1074870 (C4 (Web Text))

Max Activating Token Index: 384

Text #7

Max Range: 2.0802. Min Range: -2.0802

Max Act: 1.5611. Min Act: -0.0000

Data Index: 560906 (C4 (Web Text))

Max Activating Token Index: 429

Text #8

Max Range: 2.0802. Min Range: -2.0802

Max Act: 1.5224. Min Act: -0.0000

Data Index: 372410 (C4 (Web Text))

Max Activating Token Index: 148

Text #9

Max Range: 2.0802. Min Range: -2.0802

Max Act: 1.5633. Min Act: -0.0000

Data Index: 557383 (C4 (Web Text))

Max Activating Token Index: 518

Text #10

Max Range: 2.0802. Min Range: -2.0802

Max Act: 1.5420. Min Act: -0.0000

Data Index: 929035 (C4 (Web Text))

Max Activating Token Index: 717

Text #11

Max Range: 2.0802. Min Range: -2.0802

Max Act: 1.3901. Min Act: -0.0000

Data Index: 93838 (C4 (Web Text))

Max Activating Token Index: 563

Text #12

Max Range: 2.0802. Min Range: -2.0802

Max Act: 1.3055. Min Act: -0.0000

Data Index: 135343 (C4 (Web Text))

Max Activating Token Index: 267

Text #13

Max Range: 2.0802. Min Range: -2.0802

Max Act: 1.2726. Min Act: -0.0000

Data Index: 253106 (C4 (Web Text))

Max Activating Token Index: 487

Text #14

Max Range: 2.0802. Min Range: -2.0802

Max Act: 1.3647. Min Act: -0.0000

Data Index: 21029 (C4 (Web Text))

Max Activating Token Index: 352

Text #15

Max Range: 2.0802. Min Range: -2.0802

Max Act: 1.3518. Min Act: -0.0000

Data Index: 626368 (C4 (Web Text))

Max Activating Token Index: 769

Text #16

Max Range: 2.0802. Min Range: -2.0802

Max Act: 1.2534. Min Act: -0.0000

Data Index: 779638 (C4 (Web Text))

Max Activating Token Index: 960

Text #17

Max Range: 2.0802. Min Range: -2.0802

Max Act: 1.1090. Min Act: -0.0000

Data Index: 1265337 (C4 (Web Text))

Max Activating Token Index: 133

Text #18

Max Range: 2.0802. Min Range: -2.0802

Max Act: 1.1818. Min Act: -0.0000

Data Index: 643513 (C4 (Web Text))

Max Activating Token Index: 1018

Text #19

Max Range: 2.0802. Min Range: -2.0802

Max Act: 1.1241. Min Act: -0.0000

Data Index: 1166552 (C4 (Web Text))

Max Activating Token Index: 340