Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2534 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.4763. Min Range: -0.4763

Max Act: 0.4698. Min Act: -0.0000

Data Index: 1617373 (Python Code)

Max Activating Token Index: 613

Text #1

Max Range: 0.4763. Min Range: -0.4763

Max Act: 0.4763. Min Act: -0.0000

Data Index: 1367342 (Python Code)

Max Activating Token Index: 718

Text #2

Max Range: 0.4763. Min Range: -0.4763

Max Act: 0.4535. Min Act: -0.0000

Data Index: 1656165 (Python Code)

Max Activating Token Index: 698

Text #3

Max Range: 0.4763. Min Range: -0.4763

Max Act: 0.4336. Min Act: -0.0000

Data Index: 1396025 (Python Code)

Max Activating Token Index: 701

Text #4

Max Range: 0.4763. Min Range: -0.4763

Max Act: 0.4481. Min Act: -0.0000

Data Index: 1600567 (Python Code)

Max Activating Token Index: 842

Text #5

Max Range: 0.4763. Min Range: -0.4763

Max Act: 0.4330. Min Act: -0.0000

Data Index: 1596586 (Python Code)

Max Activating Token Index: 889

Text #6

Max Range: 0.4763. Min Range: -0.4763

Max Act: 0.4361. Min Act: -0.0000

Data Index: 1501769 (Python Code)

Max Activating Token Index: 187

Text #7

Max Range: 0.4763. Min Range: -0.4763

Max Act: 0.4343. Min Act: -0.0000

Data Index: 1552180 (Python Code)

Max Activating Token Index: 600

Text #8

Max Range: 0.4763. Min Range: -0.4763

Max Act: 0.3854. Min Act: -0.0000

Data Index: 1391385 (Python Code)

Max Activating Token Index: 551

Text #9

Max Range: 0.4763. Min Range: -0.4763

Max Act: 0.4150. Min Act: -0.0000

Data Index: 1389266 (Python Code)

Max Activating Token Index: 766

Text #10

Max Range: 0.4763. Min Range: -0.4763

Max Act: 0.4173. Min Act: -0.0000

Data Index: 1640255 (Python Code)

Max Activating Token Index: 508

Text #11

Max Range: 0.4763. Min Range: -0.4763

Max Act: 0.3920. Min Act: -0.0000

Data Index: 1616471 (Python Code)

Max Activating Token Index: 291

Text #12

Max Range: 0.4763. Min Range: -0.4763

Max Act: 0.3900. Min Act: -0.0000

Data Index: 1395727 (Python Code)

Max Activating Token Index: 86

Text #13

Max Range: 0.4763. Min Range: -0.4763

Max Act: 0.3613. Min Act: -0.0000

Data Index: 1509761 (Python Code)

Max Activating Token Index: 881

Text #14

Max Range: 0.4763. Min Range: -0.4763

Max Act: 0.3607. Min Act: -0.0000

Data Index: 1379781 (Python Code)

Max Activating Token Index: 890

Text #15

Max Range: 0.4763. Min Range: -0.4763

Max Act: 0.3546. Min Act: -0.0000

Data Index: 1653568 (Python Code)

Max Activating Token Index: 293

Text #16

Max Range: 0.4763. Min Range: -0.4763

Max Act: 0.3592. Min Act: -0.0000

Data Index: 1632812 (Python Code)

Max Activating Token Index: 584

Text #17

Max Range: 0.4763. Min Range: -0.4763

Max Act: 0.3467. Min Act: -0.0000

Data Index: 1448428 (Python Code)

Max Activating Token Index: 682

Text #18

Max Range: 0.4763. Min Range: -0.4763

Max Act: 0.3834. Min Act: -0.0000

Data Index: 1431626 (Python Code)

Max Activating Token Index: 578

Text #19

Max Range: 0.4763. Min Range: -0.4763

Max Act: 0.3572. Min Act: -0.0000

Data Index: 1552578 (Python Code)

Max Activating Token Index: 195