Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 406 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 2.2604. Min Range: -2.2604

Max Act: 2.2604. Min Act: -0.0000

Data Index: 699514 (C4 (Web Text))

Max Activating Token Index: 11

Text #1

Max Range: 2.2604. Min Range: -2.2604

Max Act: 2.0831. Min Act: -0.0000

Data Index: 401218 (C4 (Web Text))

Max Activating Token Index: 692

Text #2

Max Range: 2.2604. Min Range: -2.2604

Max Act: 1.7437. Min Act: -0.0000

Data Index: 1150168 (C4 (Web Text))

Max Activating Token Index: 444

Text #3

Max Range: 2.2604. Min Range: -2.2604

Max Act: 1.7323. Min Act: -0.0000

Data Index: 560814 (C4 (Web Text))

Max Activating Token Index: 255

Text #4

Max Range: 2.2604. Min Range: -2.2604

Max Act: 1.6823. Min Act: -0.0000

Data Index: 596564 (C4 (Web Text))

Max Activating Token Index: 884

Text #5

Max Range: 2.2604. Min Range: -2.2604

Max Act: 1.7668. Min Act: -0.0000

Data Index: 1080283 (C4 (Web Text))

Max Activating Token Index: 10

Text #6

Max Range: 2.2604. Min Range: -2.2604

Max Act: 1.6614. Min Act: -0.0000

Data Index: 669727 (C4 (Web Text))

Max Activating Token Index: 875

Text #7

Max Range: 2.2604. Min Range: -2.2604

Max Act: 1.7589. Min Act: -0.0000

Data Index: 434150 (C4 (Web Text))

Max Activating Token Index: 964

Text #8

Max Range: 2.2604. Min Range: -2.2604

Max Act: 1.6650. Min Act: -0.0000

Data Index: 870922 (C4 (Web Text))

Max Activating Token Index: 765

Text #9

Max Range: 2.2604. Min Range: -2.2604

Max Act: 1.5043. Min Act: -0.0000

Data Index: 700111 (C4 (Web Text))

Max Activating Token Index: 862

Text #10

Max Range: 2.2604. Min Range: -2.2604

Max Act: 1.6859. Min Act: -0.0000

Data Index: 1311335 (C4 (Web Text))

Max Activating Token Index: 513

Text #11

Max Range: 2.2604. Min Range: -2.2604

Max Act: 1.7322. Min Act: -0.0000

Data Index: 1053045 (C4 (Web Text))

Max Activating Token Index: 657

Text #12

Max Range: 2.2604. Min Range: -2.2604

Max Act: 1.5937. Min Act: -0.0000

Data Index: 15905 (C4 (Web Text))

Max Activating Token Index: 726

Text #13

Max Range: 2.2604. Min Range: -2.2604

Max Act: 1.6465. Min Act: -0.0000

Data Index: 147023 (C4 (Web Text))

Max Activating Token Index: 724

Text #14

Max Range: 2.2604. Min Range: -2.2604

Max Act: 1.6518. Min Act: -0.0000

Data Index: 1215103 (C4 (Web Text))

Max Activating Token Index: 580

Text #15

Max Range: 2.2604. Min Range: -2.2604

Max Act: 1.4552. Min Act: -0.0000

Data Index: 101310 (C4 (Web Text))

Max Activating Token Index: 310

Text #16

Max Range: 2.2604. Min Range: -2.2604

Max Act: 1.6601. Min Act: -0.0000

Data Index: 453265 (C4 (Web Text))

Max Activating Token Index: 367

Text #17

Max Range: 2.2604. Min Range: -2.2604

Max Act: 1.5917. Min Act: -0.0000

Data Index: 889953 (C4 (Web Text))

Max Activating Token Index: 405

Text #18

Max Range: 2.2604. Min Range: -2.2604

Max Act: 1.4995. Min Act: -0.0000

Data Index: 35641 (C4 (Web Text))

Max Activating Token Index: 333

Text #19

Max Range: 2.2604. Min Range: -2.2604

Max Act: 1.6254. Min Act: -0.0000

Data Index: 313079 (C4 (Web Text))

Max Activating Token Index: 989