Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4483 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.1958. Min Range: -0.1958

Max Act: 0.1958. Min Act: -0.0000

Data Index: 483055 (C4 (Web Text))

Max Activating Token Index: 460

Text #1

Max Range: 0.1958. Min Range: -0.1958

Max Act: 0.1955. Min Act: -0.0000

Data Index: 154743 (C4 (Web Text))

Max Activating Token Index: 505

Text #2

Max Range: 0.1958. Min Range: -0.1958

Max Act: 0.1929. Min Act: -0.0000

Data Index: 847708 (C4 (Web Text))

Max Activating Token Index: 633

Text #3

Max Range: 0.1958. Min Range: -0.1958

Max Act: 0.1814. Min Act: -0.0000

Data Index: 412558 (C4 (Web Text))

Max Activating Token Index: 364

Text #4

Max Range: 0.1958. Min Range: -0.1958

Max Act: 0.1704. Min Act: -0.0000

Data Index: 166133 (C4 (Web Text))

Max Activating Token Index: 255

Text #5

Max Range: 0.1958. Min Range: -0.1958

Max Act: 0.1657. Min Act: -0.0000

Data Index: 64678 (C4 (Web Text))

Max Activating Token Index: 110

Text #6

Max Range: 0.1958. Min Range: -0.1958

Max Act: 0.1624. Min Act: -0.0000

Data Index: 600650 (C4 (Web Text))

Max Activating Token Index: 508

Text #7

Max Range: 0.1958. Min Range: -0.1958

Max Act: 0.1597. Min Act: -0.0000

Data Index: 220967 (C4 (Web Text))

Max Activating Token Index: 65

Text #8

Max Range: 0.1958. Min Range: -0.1958

Max Act: 0.1587. Min Act: -0.0000

Data Index: 1311802 (C4 (Web Text))

Max Activating Token Index: 762

Text #9

Max Range: 0.1958. Min Range: -0.1958

Max Act: 0.1576. Min Act: -0.0000

Data Index: 729341 (C4 (Web Text))

Max Activating Token Index: 637

Text #10

Max Range: 0.1958. Min Range: -0.1958

Max Act: 0.1582. Min Act: -0.0000

Data Index: 849310 (C4 (Web Text))

Max Activating Token Index: 797

Text #11

Max Range: 0.1958. Min Range: -0.1958

Max Act: 0.1499. Min Act: -0.0000

Data Index: 240228 (C4 (Web Text))

Max Activating Token Index: 580

Text #12

Max Range: 0.1958. Min Range: -0.1958

Max Act: 0.1441. Min Act: -0.0000

Data Index: 755880 (C4 (Web Text))

Max Activating Token Index: 711

Text #13

Max Range: 0.1958. Min Range: -0.1958

Max Act: 0.1460. Min Act: -0.0000

Data Index: 1167279 (C4 (Web Text))

Max Activating Token Index: 855

Text #14

Max Range: 0.1958. Min Range: -0.1958

Max Act: 0.1438. Min Act: -0.0000

Data Index: 729568 (C4 (Web Text))

Max Activating Token Index: 683

Text #15

Max Range: 0.1958. Min Range: -0.1958

Max Act: 0.1490. Min Act: -0.0000

Data Index: 608066 (C4 (Web Text))

Max Activating Token Index: 679

Text #16

Max Range: 0.1958. Min Range: -0.1958

Max Act: 0.1392. Min Act: -0.0000

Data Index: 1270637 (C4 (Web Text))

Max Activating Token Index: 138

Text #17

Max Range: 0.1958. Min Range: -0.1958

Max Act: 0.1464. Min Act: -0.0000

Data Index: 867952 (C4 (Web Text))

Max Activating Token Index: 485

Text #18

Max Range: 0.1958. Min Range: -0.1958

Max Act: 0.1418. Min Act: -0.0000

Data Index: 444358 (C4 (Web Text))

Max Activating Token Index: 479

Text #19

Max Range: 0.1958. Min Range: -0.1958

Max Act: 0.1343. Min Act: -0.0000

Data Index: 120124 (C4 (Web Text))

Max Activating Token Index: 640