Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2043 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.8570. Min Range: -0.8570

Max Act: 0.8507. Min Act: -0.0000

Data Index: 1401643 (Python Code)

Max Activating Token Index: 222

Text #1

Max Range: 0.8570. Min Range: -0.8570

Max Act: 0.8570. Min Act: -0.0000

Data Index: 574486 (C4 (Web Text))

Max Activating Token Index: 943

Text #2

Max Range: 0.8570. Min Range: -0.8570

Max Act: 0.7328. Min Act: -0.0000

Data Index: 1423207 (Python Code)

Max Activating Token Index: 386

Text #3

Max Range: 0.8570. Min Range: -0.8570

Max Act: 0.6706. Min Act: -0.0000

Data Index: 864381 (C4 (Web Text))

Max Activating Token Index: 610

Text #4

Max Range: 0.8570. Min Range: -0.8570

Max Act: 0.6530. Min Act: -0.0000

Data Index: 1453641 (Python Code)

Max Activating Token Index: 236

Text #5

Max Range: 0.8570. Min Range: -0.8570

Max Act: 0.6261. Min Act: -0.0000

Data Index: 1442197 (Python Code)

Max Activating Token Index: 104

Text #6

Max Range: 0.8570. Min Range: -0.8570

Max Act: 0.7131. Min Act: -0.0000

Data Index: 1625338 (Python Code)

Max Activating Token Index: 904

Text #7

Max Range: 0.8570. Min Range: -0.8570

Max Act: 0.6455. Min Act: -0.0000

Data Index: 1609689 (Python Code)

Max Activating Token Index: 586

Text #8

Max Range: 0.8570. Min Range: -0.8570

Max Act: 0.6135. Min Act: -0.0000

Data Index: 1542200 (Python Code)

Max Activating Token Index: 32

Text #9

Max Range: 0.8570. Min Range: -0.8570

Max Act: 0.6548. Min Act: -0.0000

Data Index: 1568971 (Python Code)

Max Activating Token Index: 155

Text #10

Max Range: 0.8570. Min Range: -0.8570

Max Act: 0.5586. Min Act: -0.0000

Data Index: 1507281 (Python Code)

Max Activating Token Index: 307

Text #11

Max Range: 0.8570. Min Range: -0.8570

Max Act: 0.6527. Min Act: -0.0000

Data Index: 1325136 (C4 (Web Text))

Max Activating Token Index: 908

Text #12

Max Range: 0.8570. Min Range: -0.8570

Max Act: 0.6587. Min Act: -0.0000

Data Index: 1392142 (Python Code)

Max Activating Token Index: 755

Text #13

Max Range: 0.8570. Min Range: -0.8570

Max Act: 0.5916. Min Act: -0.0000

Data Index: 1377685 (Python Code)

Max Activating Token Index: 733

Text #14

Max Range: 0.8570. Min Range: -0.8570

Max Act: 0.6361. Min Act: -0.0000

Data Index: 1654457 (Python Code)

Max Activating Token Index: 252

Text #15

Max Range: 0.8570. Min Range: -0.8570

Max Act: 0.5908. Min Act: -0.0000

Data Index: 1458695 (Python Code)

Max Activating Token Index: 472

Text #16

Max Range: 0.8570. Min Range: -0.8570

Max Act: 0.6062. Min Act: -0.0000

Data Index: 1479890 (Python Code)

Max Activating Token Index: 948

Text #17

Max Range: 0.8570. Min Range: -0.8570

Max Act: 0.5685. Min Act: -0.0000

Data Index: 1602641 (Python Code)

Max Activating Token Index: 873

Text #18

Max Range: 0.8570. Min Range: -0.8570

Max Act: 0.5521. Min Act: -0.0000

Data Index: 1578631 (Python Code)

Max Activating Token Index: 248

Text #19

Max Range: 0.8570. Min Range: -0.8570

Max Act: 0.5186. Min Act: -0.0000

Data Index: 1580817 (Python Code)

Max Activating Token Index: 612