Model: SoLU Model: 8 Layers, 4096 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1252 in Layer 1

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-8l')

Text #0

Max Range: 0.0795. Min Range: -0.0795

Max Act: 0.0775. Min Act: -0.0003

Data Index: 1281640 (C4 (Web Text))

Max Activating Token Index: 365

Text #1

Max Range: 0.0795. Min Range: -0.0795

Max Act: 0.0795. Min Act: -0.0003

Data Index: 209644 (C4 (Web Text))

Max Activating Token Index: 33

Text #2

Max Range: 0.0795. Min Range: -0.0795

Max Act: 0.0729. Min Act: -0.0003

Data Index: 667947 (C4 (Web Text))

Max Activating Token Index: 418

Text #3

Max Range: 0.0795. Min Range: -0.0795

Max Act: 0.0736. Min Act: -0.0003

Data Index: 188059 (C4 (Web Text))

Max Activating Token Index: 919

Text #4

Max Range: 0.0795. Min Range: -0.0795

Max Act: 0.0712. Min Act: -0.0003

Data Index: 162551 (C4 (Web Text))

Max Activating Token Index: 68

Text #5

Max Range: 0.0795. Min Range: -0.0795

Max Act: 0.0722. Min Act: -0.0003

Data Index: 529142 (C4 (Web Text))

Max Activating Token Index: 453

Text #6

Max Range: 0.0795. Min Range: -0.0795

Max Act: 0.0723. Min Act: -0.0003

Data Index: 499424 (C4 (Web Text))

Max Activating Token Index: 196

Text #7

Max Range: 0.0795. Min Range: -0.0795

Max Act: 0.0713. Min Act: -0.0003

Data Index: 1274160 (C4 (Web Text))

Max Activating Token Index: 384

Text #8

Max Range: 0.0795. Min Range: -0.0795

Max Act: 0.0740. Min Act: -0.0003

Data Index: 828505 (C4 (Web Text))

Max Activating Token Index: 912

Text #9

Max Range: 0.0795. Min Range: -0.0795

Max Act: 0.0681. Min Act: -0.0003

Data Index: 1086684 (C4 (Web Text))

Max Activating Token Index: 551

Text #10

Max Range: 0.0795. Min Range: -0.0795

Max Act: 0.0688. Min Act: -0.0003

Data Index: 447626 (C4 (Web Text))

Max Activating Token Index: 534

Text #11

Max Range: 0.0795. Min Range: -0.0795

Max Act: 0.0707. Min Act: -0.0003

Data Index: 285431 (C4 (Web Text))

Max Activating Token Index: 576

Text #12

Max Range: 0.0795. Min Range: -0.0795

Max Act: 0.0683. Min Act: -0.0003

Data Index: 665054 (C4 (Web Text))

Max Activating Token Index: 416

Text #13

Max Range: 0.0795. Min Range: -0.0795

Max Act: 0.0730. Min Act: -0.0003

Data Index: 453826 (C4 (Web Text))

Max Activating Token Index: 549

Text #14

Max Range: 0.0795. Min Range: -0.0795

Max Act: 0.0700. Min Act: -0.0003

Data Index: 1293275 (C4 (Web Text))

Max Activating Token Index: 966

Text #15

Max Range: 0.0795. Min Range: -0.0795

Max Act: 0.0651. Min Act: -0.0003

Data Index: 1269323 (C4 (Web Text))

Max Activating Token Index: 760

Text #16

Max Range: 0.0795. Min Range: -0.0795

Max Act: 0.0694. Min Act: -0.0003

Data Index: 619424 (C4 (Web Text))

Max Activating Token Index: 313

Text #17

Max Range: 0.0795. Min Range: -0.0795

Max Act: 0.0674. Min Act: -0.0003

Data Index: 1344467 (C4 (Web Text))

Max Activating Token Index: 326

Text #18

Max Range: 0.0795. Min Range: -0.0795

Max Act: 0.0694. Min Act: -0.0003

Data Index: 749509 (C4 (Web Text))

Max Activating Token Index: 963

Text #19

Max Range: 0.0795. Min Range: -0.0795

Max Act: 0.0663. Min Act: -0.0003

Data Index: 239933 (C4 (Web Text))

Max Activating Token Index: 581