Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2460 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.1686. Min Range: -1.1686

Max Act: 1.1686. Min Act: -0.0000

Data Index: 40383 (C4 (Web Text))

Max Activating Token Index: 831

Text #1

Max Range: 1.1686. Min Range: -1.1686

Max Act: 0.9414. Min Act: -0.0000

Data Index: 300533 (C4 (Web Text))

Max Activating Token Index: 962

Text #2

Max Range: 1.1686. Min Range: -1.1686

Max Act: 0.7506. Min Act: -0.0000

Data Index: 607851 (C4 (Web Text))

Max Activating Token Index: 565

Text #3

Max Range: 1.1686. Min Range: -1.1686

Max Act: 0.7736. Min Act: -0.0000

Data Index: 458056 (C4 (Web Text))

Max Activating Token Index: 1009

Text #4

Max Range: 1.1686. Min Range: -1.1686

Max Act: 0.7253. Min Act: -0.0000

Data Index: 361133 (C4 (Web Text))

Max Activating Token Index: 986

Text #5

Max Range: 1.1686. Min Range: -1.1686

Max Act: 0.6185. Min Act: -0.0000

Data Index: 1133593 (C4 (Web Text))

Max Activating Token Index: 760

Text #6

Max Range: 1.1686. Min Range: -1.1686

Max Act: 0.6398. Min Act: -0.0000

Data Index: 708299 (C4 (Web Text))

Max Activating Token Index: 272

Text #7

Max Range: 1.1686. Min Range: -1.1686

Max Act: 0.6108. Min Act: -0.0000

Data Index: 1267197 (C4 (Web Text))

Max Activating Token Index: 567

Text #8

Max Range: 1.1686. Min Range: -1.1686

Max Act: 0.6467. Min Act: -0.0000

Data Index: 111416 (C4 (Web Text))

Max Activating Token Index: 376

Text #9

Max Range: 1.1686. Min Range: -1.1686

Max Act: 0.6366. Min Act: -0.0000

Data Index: 564581 (C4 (Web Text))

Max Activating Token Index: 432

Text #10

Max Range: 1.1686. Min Range: -1.1686

Max Act: 0.5762. Min Act: -0.0000

Data Index: 899149 (C4 (Web Text))

Max Activating Token Index: 883

Text #11

Max Range: 1.1686. Min Range: -1.1686

Max Act: 0.5944. Min Act: -0.0000

Data Index: 870252 (C4 (Web Text))

Max Activating Token Index: 460

Text #12

Max Range: 1.1686. Min Range: -1.1686

Max Act: 0.5442. Min Act: -0.0000

Data Index: 585785 (C4 (Web Text))

Max Activating Token Index: 844

Text #13

Max Range: 1.1686. Min Range: -1.1686

Max Act: 0.5815. Min Act: -0.0000

Data Index: 1357247 (C4 (Web Text))

Max Activating Token Index: 275

Text #14

Max Range: 1.1686. Min Range: -1.1686

Max Act: 0.5762. Min Act: -0.0000

Data Index: 267887 (C4 (Web Text))

Max Activating Token Index: 523

Text #15

Max Range: 1.1686. Min Range: -1.1686

Max Act: 0.5812. Min Act: -0.0000

Data Index: 1103833 (C4 (Web Text))

Max Activating Token Index: 108

Text #16

Max Range: 1.1686. Min Range: -1.1686

Max Act: 0.5618. Min Act: -0.0000

Data Index: 2104 (C4 (Web Text))

Max Activating Token Index: 887

Text #17

Max Range: 1.1686. Min Range: -1.1686

Max Act: 0.5420. Min Act: -0.0000

Data Index: 446625 (C4 (Web Text))

Max Activating Token Index: 881

Text #18

Max Range: 1.1686. Min Range: -1.1686

Max Act: 0.5418. Min Act: -0.0000

Data Index: 132882 (C4 (Web Text))

Max Activating Token Index: 976

Text #19

Max Range: 1.1686. Min Range: -1.1686

Max Act: 0.5748. Min Act: -0.0000

Data Index: 122985 (C4 (Web Text))

Max Activating Token Index: 509