Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 531 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 2.1875. Min Range: -2.1875

Max Act: 2.0877. Min Act: -0.0000

Data Index: 1649415 (Python Code)

Max Activating Token Index: 106

Text #1

Max Range: 2.1875. Min Range: -2.1875

Max Act: 1.8905. Min Act: -0.0000

Data Index: 1468129 (Python Code)

Max Activating Token Index: 897

Text #2

Max Range: 2.1875. Min Range: -2.1875

Max Act: 1.9840. Min Act: -0.0000

Data Index: 1502407 (Python Code)

Max Activating Token Index: 391

Text #3

Max Range: 2.1875. Min Range: -2.1875

Max Act: 1.8661. Min Act: -0.0000

Data Index: 1569526 (Python Code)

Max Activating Token Index: 230

Text #4

Max Range: 2.1875. Min Range: -2.1875

Max Act: 1.9069. Min Act: -0.0000

Data Index: 1366501 (Python Code)

Max Activating Token Index: 450

Text #5

Max Range: 2.1875. Min Range: -2.1875

Max Act: 1.8869. Min Act: -0.0000

Data Index: 1631007 (Python Code)

Max Activating Token Index: 481

Text #6

Max Range: 2.1875. Min Range: -2.1875

Max Act: 2.1875. Min Act: -0.0000

Data Index: 1373286 (Python Code)

Max Activating Token Index: 301

Text #7

Max Range: 2.1875. Min Range: -2.1875

Max Act: 1.9124. Min Act: -0.0000

Data Index: 1545361 (Python Code)

Max Activating Token Index: 623

Text #8

Max Range: 2.1875. Min Range: -2.1875

Max Act: 1.6663. Min Act: -0.0000

Data Index: 1555395 (Python Code)

Max Activating Token Index: 478

Text #9

Max Range: 2.1875. Min Range: -2.1875

Max Act: 1.8240. Min Act: -0.0000

Data Index: 1593288 (Python Code)

Max Activating Token Index: 620

Text #10

Max Range: 2.1875. Min Range: -2.1875

Max Act: 1.5316. Min Act: -0.0000

Data Index: 1590063 (Python Code)

Max Activating Token Index: 91

Text #11

Max Range: 2.1875. Min Range: -2.1875

Max Act: 1.5215. Min Act: -0.0000

Data Index: 1569324 (Python Code)

Max Activating Token Index: 799

Text #12

Max Range: 2.1875. Min Range: -2.1875

Max Act: 1.5023. Min Act: -0.0000

Data Index: 1615076 (Python Code)

Max Activating Token Index: 804

Text #13

Max Range: 2.1875. Min Range: -2.1875

Max Act: 1.5793. Min Act: -0.0000

Data Index: 1393775 (Python Code)

Max Activating Token Index: 171

Text #14

Max Range: 2.1875. Min Range: -2.1875

Max Act: 1.5918. Min Act: -0.0000

Data Index: 1366742 (Python Code)

Max Activating Token Index: 955

Text #15

Max Range: 2.1875. Min Range: -2.1875

Max Act: 1.3960. Min Act: -0.0000

Data Index: 1509591 (Python Code)

Max Activating Token Index: 211

Text #16

Max Range: 2.1875. Min Range: -2.1875

Max Act: 1.5620. Min Act: -0.0000

Data Index: 1401790 (Python Code)

Max Activating Token Index: 198

Text #17

Max Range: 2.1875. Min Range: -2.1875

Max Act: 1.4714. Min Act: -0.0000

Data Index: 1411246 (Python Code)

Max Activating Token Index: 844

Text #18

Max Range: 2.1875. Min Range: -2.1875

Max Act: 1.5542. Min Act: -0.0000

Data Index: 1534155 (Python Code)

Max Activating Token Index: 365

Text #19

Max Range: 2.1875. Min Range: -2.1875

Max Act: 1.5184. Min Act: -0.0000

Data Index: 1490985 (Python Code)

Max Activating Token Index: 272