Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4062 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.7784. Min Range: -0.7784

Max Act: 0.7591. Min Act: 0.0001

Data Index: 1558402 (Python Code)

Max Activating Token Index: 1017

Text #1

Max Range: 0.7784. Min Range: -0.7784

Max Act: 0.7784. Min Act: 0.0001

Data Index: 1486554 (Python Code)

Max Activating Token Index: 753

Text #2

Max Range: 0.7784. Min Range: -0.7784

Max Act: 0.6745. Min Act: 0.0001

Data Index: 1393120 (Python Code)

Max Activating Token Index: 604

Text #3

Max Range: 0.7784. Min Range: -0.7784

Max Act: 0.6513. Min Act: 0.0001

Data Index: 1471146 (Python Code)

Max Activating Token Index: 996

Text #4

Max Range: 0.7784. Min Range: -0.7784

Max Act: 0.6127. Min Act: -0.0000

Data Index: 1603998 (Python Code)

Max Activating Token Index: 847

Text #5

Max Range: 0.7784. Min Range: -0.7784

Max Act: 0.5814. Min Act: 0.0002

Data Index: 1366433 (Python Code)

Max Activating Token Index: 641

Text #6

Max Range: 0.7784. Min Range: -0.7784

Max Act: 0.5912. Min Act: 0.0001

Data Index: 1562376 (Python Code)

Max Activating Token Index: 619

Text #7

Max Range: 0.7784. Min Range: -0.7784

Max Act: 0.6004. Min Act: 0.0000

Data Index: 1627721 (Python Code)

Max Activating Token Index: 306

Text #8

Max Range: 0.7784. Min Range: -0.7784

Max Act: 0.5518. Min Act: -0.0000

Data Index: 1607733 (Python Code)

Max Activating Token Index: 910

Text #9

Max Range: 0.7784. Min Range: -0.7784

Max Act: 0.5359. Min Act: -0.0000

Data Index: 1486180 (Python Code)

Max Activating Token Index: 203

Text #10

Max Range: 0.7784. Min Range: -0.7784

Max Act: 0.5402. Min Act: -0.0000

Data Index: 1616321 (Python Code)

Max Activating Token Index: 760

Text #11

Max Range: 0.7784. Min Range: -0.7784

Max Act: 0.5539. Min Act: 0.0001

Data Index: 1379368 (Python Code)

Max Activating Token Index: 621

Text #12

Max Range: 0.7784. Min Range: -0.7784

Max Act: 0.5174. Min Act: -0.0000

Data Index: 1638555 (Python Code)

Max Activating Token Index: 473

Text #13

Max Range: 0.7784. Min Range: -0.7784

Max Act: 0.5260. Min Act: 0.0002

Data Index: 1521904 (Python Code)

Max Activating Token Index: 68

Text #14

Max Range: 0.7784. Min Range: -0.7784

Max Act: 0.4856. Min Act: 0.0000

Data Index: 1539262 (Python Code)

Max Activating Token Index: 610

Text #15

Max Range: 0.7784. Min Range: -0.7784

Max Act: 0.4758. Min Act: -0.0000

Data Index: 1656241 (Python Code)

Max Activating Token Index: 174

Text #16

Max Range: 0.7784. Min Range: -0.7784

Max Act: 0.4782. Min Act: -0.0000

Data Index: 1367250 (Python Code)

Max Activating Token Index: 283

Text #17

Max Range: 0.7784. Min Range: -0.7784

Max Act: 0.4795. Min Act: 0.0002

Data Index: 1526730 (Python Code)

Max Activating Token Index: 43

Text #18

Max Range: 0.7784. Min Range: -0.7784

Max Act: 0.4864. Min Act: 0.0002

Data Index: 1407187 (Python Code)

Max Activating Token Index: 315

Text #19

Max Range: 0.7784. Min Range: -0.7784

Max Act: 0.4767. Min Act: 0.0000

Data Index: 1601534 (Python Code)

Max Activating Token Index: 517