Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3587 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.9948. Min Range: -0.9948

Max Act: 0.9752. Min Act: -0.0000

Data Index: 1490286 (Python Code)

Max Activating Token Index: 989

Text #1

Max Range: 0.9948. Min Range: -0.9948

Max Act: 0.9948. Min Act: -0.0000

Data Index: 1455897 (Python Code)

Max Activating Token Index: 987

Text #2

Max Range: 0.9948. Min Range: -0.9948

Max Act: 0.7936. Min Act: -0.0000

Data Index: 1579760 (Python Code)

Max Activating Token Index: 855

Text #3

Max Range: 0.9948. Min Range: -0.9948

Max Act: 0.8086. Min Act: -0.0000

Data Index: 1502133 (Python Code)

Max Activating Token Index: 708

Text #4

Max Range: 0.9948. Min Range: -0.9948

Max Act: 0.8114. Min Act: -0.0000

Data Index: 1577272 (Python Code)

Max Activating Token Index: 716

Text #5

Max Range: 0.9948. Min Range: -0.9948

Max Act: 0.7760. Min Act: -0.0000

Data Index: 1591656 (Python Code)

Max Activating Token Index: 665

Text #6

Max Range: 0.9948. Min Range: -0.9948

Max Act: 0.7763. Min Act: -0.0000

Data Index: 1623376 (Python Code)

Max Activating Token Index: 965

Text #7

Max Range: 0.9948. Min Range: -0.9948

Max Act: 0.8046. Min Act: -0.0000

Data Index: 1571595 (Python Code)

Max Activating Token Index: 448

Text #8

Max Range: 0.9948. Min Range: -0.9948

Max Act: 0.7471. Min Act: -0.0000

Data Index: 1623498 (Python Code)

Max Activating Token Index: 255

Text #9

Max Range: 0.9948. Min Range: -0.9948

Max Act: 0.7460. Min Act: -0.0000

Data Index: 1636535 (Python Code)

Max Activating Token Index: 748

Text #10

Max Range: 0.9948. Min Range: -0.9948

Max Act: 0.7458. Min Act: -0.0000

Data Index: 1518296 (Python Code)

Max Activating Token Index: 589

Text #11

Max Range: 0.9948. Min Range: -0.9948

Max Act: 0.7443. Min Act: -0.0000

Data Index: 1395791 (Python Code)

Max Activating Token Index: 857

Text #12

Max Range: 0.9948. Min Range: -0.9948

Max Act: 0.6730. Min Act: -0.0000

Data Index: 1474934 (Python Code)

Max Activating Token Index: 627

Text #13

Max Range: 0.9948. Min Range: -0.9948

Max Act: 0.6936. Min Act: -0.0000

Data Index: 1484550 (Python Code)

Max Activating Token Index: 816

Text #14

Max Range: 0.9948. Min Range: -0.9948

Max Act: 0.6699. Min Act: -0.0000

Data Index: 1515754 (Python Code)

Max Activating Token Index: 151

Text #15

Max Range: 0.9948. Min Range: -0.9948

Max Act: 0.7323. Min Act: -0.0000

Data Index: 1566170 (Python Code)

Max Activating Token Index: 885

Text #16

Max Range: 0.9948. Min Range: -0.9948

Max Act: 0.7808. Min Act: -0.0000

Data Index: 1440775 (Python Code)

Max Activating Token Index: 914

Text #17

Max Range: 0.9948. Min Range: -0.9948

Max Act: 0.7431. Min Act: -0.0000

Data Index: 1574903 (Python Code)

Max Activating Token Index: 454

Text #18

Max Range: 0.9948. Min Range: -0.9948

Max Act: 0.7655. Min Act: -0.0000

Data Index: 1544262 (Python Code)

Max Activating Token Index: 629

Text #19

Max Range: 0.9948. Min Range: -0.9948

Max Act: 0.6170. Min Act: -0.0000

Data Index: 1384157 (Python Code)

Max Activating Token Index: 778