Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 248 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.5720. Min Range: -1.5720

Max Act: 1.5720. Min Act: -0.0000

Data Index: 1400112 (Python Code)

Max Activating Token Index: 320

Text #1

Max Range: 1.5720. Min Range: -1.5720

Max Act: 1.4023. Min Act: -0.0000

Data Index: 1531667 (Python Code)

Max Activating Token Index: 585

Text #2

Max Range: 1.5720. Min Range: -1.5720

Max Act: 1.0313. Min Act: -0.0000

Data Index: 1635612 (Python Code)

Max Activating Token Index: 575

Text #3

Max Range: 1.5720. Min Range: -1.5720

Max Act: 1.0393. Min Act: -0.0000

Data Index: 1362306 (Python Code)

Max Activating Token Index: 934

Text #4

Max Range: 1.5720. Min Range: -1.5720

Max Act: 0.9414. Min Act: -0.0000

Data Index: 1543577 (Python Code)

Max Activating Token Index: 553

Text #5

Max Range: 1.5720. Min Range: -1.5720

Max Act: 0.8847. Min Act: -0.0000

Data Index: 1451849 (Python Code)

Max Activating Token Index: 648

Text #6

Max Range: 1.5720. Min Range: -1.5720

Max Act: 0.8840. Min Act: -0.0000

Data Index: 1493817 (Python Code)

Max Activating Token Index: 965

Text #7

Max Range: 1.5720. Min Range: -1.5720

Max Act: 0.8789. Min Act: -0.0000

Data Index: 1532769 (Python Code)

Max Activating Token Index: 892

Text #8

Max Range: 1.5720. Min Range: -1.5720

Max Act: 0.8780. Min Act: -0.0000

Data Index: 1566289 (Python Code)

Max Activating Token Index: 417

Text #9

Max Range: 1.5720. Min Range: -1.5720

Max Act: 0.8143. Min Act: -0.0000

Data Index: 1391995 (Python Code)

Max Activating Token Index: 420

Text #10

Max Range: 1.5720. Min Range: -1.5720

Max Act: 0.7954. Min Act: -0.0000

Data Index: 1387044 (Python Code)

Max Activating Token Index: 505

Text #11

Max Range: 1.5720. Min Range: -1.5720

Max Act: 0.8055. Min Act: -0.0000

Data Index: 1384967 (Python Code)

Max Activating Token Index: 104

Text #12

Max Range: 1.5720. Min Range: -1.5720

Max Act: 0.7890. Min Act: 0.0000

Data Index: 1584760 (Python Code)

Max Activating Token Index: 357

Text #13

Max Range: 1.5720. Min Range: -1.5720

Max Act: 0.7877. Min Act: -0.0000

Data Index: 1444911 (Python Code)

Max Activating Token Index: 812

Text #14

Max Range: 1.5720. Min Range: -1.5720

Max Act: 0.7495. Min Act: -0.0000

Data Index: 1400142 (Python Code)

Max Activating Token Index: 806

Text #15

Max Range: 1.5720. Min Range: -1.5720

Max Act: 0.6926. Min Act: -0.0000

Data Index: 1541964 (Python Code)

Max Activating Token Index: 871

Text #16

Max Range: 1.5720. Min Range: -1.5720

Max Act: 0.7787. Min Act: -0.0000

Data Index: 1395301 (Python Code)

Max Activating Token Index: 496

Text #17

Max Range: 1.5720. Min Range: -1.5720

Max Act: 0.7033. Min Act: -0.0000

Data Index: 1524107 (Python Code)

Max Activating Token Index: 930

Text #18

Max Range: 1.5720. Min Range: -1.5720

Max Act: 0.7409. Min Act: -0.0000

Data Index: 1653636 (Python Code)

Max Activating Token Index: 996

Text #19

Max Range: 1.5720. Min Range: -1.5720

Max Act: 0.7638. Min Act: -0.0000

Data Index: 1371160 (Python Code)

Max Activating Token Index: 78