Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4058 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.0711. Min Range: -1.0711

Max Act: 1.0537. Min Act: -0.0000

Data Index: 556420 (C4 (Web Text))

Max Activating Token Index: 506

Text #1

Max Range: 1.0711. Min Range: -1.0711

Max Act: 1.0711. Min Act: -0.0000

Data Index: 355060 (C4 (Web Text))

Max Activating Token Index: 56

Text #2

Max Range: 1.0711. Min Range: -1.0711

Max Act: 0.7827. Min Act: -0.0000

Data Index: 1352836 (C4 (Web Text))

Max Activating Token Index: 997

Text #3

Max Range: 1.0711. Min Range: -1.0711

Max Act: 0.7401. Min Act: -0.0000

Data Index: 682768 (C4 (Web Text))

Max Activating Token Index: 459

Text #4

Max Range: 1.0711. Min Range: -1.0711

Max Act: 0.6747. Min Act: -0.0000

Data Index: 436587 (C4 (Web Text))

Max Activating Token Index: 283

Text #5

Max Range: 1.0711. Min Range: -1.0711

Max Act: 0.7393. Min Act: -0.0000

Data Index: 8200 (C4 (Web Text))

Max Activating Token Index: 853

Text #6

Max Range: 1.0711. Min Range: -1.0711

Max Act: 0.7268. Min Act: -0.0000

Data Index: 1243834 (C4 (Web Text))

Max Activating Token Index: 9

Text #7

Max Range: 1.0711. Min Range: -1.0711

Max Act: 0.6876. Min Act: -0.0000

Data Index: 1200142 (C4 (Web Text))

Max Activating Token Index: 11

Text #8

Max Range: 1.0711. Min Range: -1.0711

Max Act: 0.6789. Min Act: -0.0000

Data Index: 10705 (C4 (Web Text))

Max Activating Token Index: 540

Text #9

Max Range: 1.0711. Min Range: -1.0711

Max Act: 0.6950. Min Act: -0.0000

Data Index: 733679 (C4 (Web Text))

Max Activating Token Index: 680

Text #10

Max Range: 1.0711. Min Range: -1.0711

Max Act: 0.6463. Min Act: -0.0000

Data Index: 1242927 (C4 (Web Text))

Max Activating Token Index: 288

Text #11

Max Range: 1.0711. Min Range: -1.0711

Max Act: 0.6636. Min Act: -0.0000

Data Index: 545886 (C4 (Web Text))

Max Activating Token Index: 862

Text #12

Max Range: 1.0711. Min Range: -1.0711

Max Act: 0.6584. Min Act: -0.0000

Data Index: 1154740 (C4 (Web Text))

Max Activating Token Index: 337

Text #13

Max Range: 1.0711. Min Range: -1.0711

Max Act: 0.6392. Min Act: -0.0000

Data Index: 1010521 (C4 (Web Text))

Max Activating Token Index: 711

Text #14

Max Range: 1.0711. Min Range: -1.0711

Max Act: 0.6119. Min Act: -0.0000

Data Index: 810336 (C4 (Web Text))

Max Activating Token Index: 746

Text #15

Max Range: 1.0711. Min Range: -1.0711

Max Act: 0.6591. Min Act: -0.0000

Data Index: 179618 (C4 (Web Text))

Max Activating Token Index: 310

Text #16

Max Range: 1.0711. Min Range: -1.0711

Max Act: 0.5802. Min Act: -0.0000

Data Index: 190544 (C4 (Web Text))

Max Activating Token Index: 189

Text #17

Max Range: 1.0711. Min Range: -1.0711

Max Act: 0.5613. Min Act: -0.0000

Data Index: 809760 (C4 (Web Text))

Max Activating Token Index: 783

Text #18

Max Range: 1.0711. Min Range: -1.0711

Max Act: 0.5780. Min Act: -0.0000

Data Index: 539826 (C4 (Web Text))

Max Activating Token Index: 837

Text #19

Max Range: 1.0711. Min Range: -1.0711

Max Act: 0.5555. Min Act: -0.0000

Data Index: 199409 (C4 (Web Text))

Max Activating Token Index: 601