Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1239 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.6077. Min Range: -0.6077

Max Act: 0.5466. Min Act: -0.0000

Data Index: 1508137 (Python Code)

Max Activating Token Index: 125

Text #1

Max Range: 0.6077. Min Range: -0.6077

Max Act: 0.6077. Min Act: -0.0000

Data Index: 1545734 (Python Code)

Max Activating Token Index: 238

Text #2

Max Range: 0.6077. Min Range: -0.6077

Max Act: 0.5399. Min Act: -0.0000

Data Index: 1647939 (Python Code)

Max Activating Token Index: 376

Text #3

Max Range: 0.6077. Min Range: -0.6077

Max Act: 0.4638. Min Act: -0.0000

Data Index: 1577725 (Python Code)

Max Activating Token Index: 136

Text #4

Max Range: 0.6077. Min Range: -0.6077

Max Act: 0.4625. Min Act: -0.0000

Data Index: 1396123 (Python Code)

Max Activating Token Index: 138

Text #5

Max Range: 0.6077. Min Range: -0.6077

Max Act: 0.4535. Min Act: -0.0000

Data Index: 1473539 (Python Code)

Max Activating Token Index: 92

Text #6

Max Range: 0.6077. Min Range: -0.6077

Max Act: 0.4452. Min Act: -0.0000

Data Index: 1505256 (Python Code)

Max Activating Token Index: 611

Text #7

Max Range: 0.6077. Min Range: -0.6077

Max Act: 0.4321. Min Act: -0.0000

Data Index: 1496381 (Python Code)

Max Activating Token Index: 328

Text #8

Max Range: 0.6077. Min Range: -0.6077

Max Act: 0.4191. Min Act: -0.0000

Data Index: 1427222 (Python Code)

Max Activating Token Index: 696

Text #9

Max Range: 0.6077. Min Range: -0.6077

Max Act: 0.3986. Min Act: -0.0000

Data Index: 1579033 (Python Code)

Max Activating Token Index: 920

Text #10

Max Range: 0.6077. Min Range: -0.6077

Max Act: 0.4005. Min Act: -0.0000

Data Index: 1510463 (Python Code)

Max Activating Token Index: 713

Text #11

Max Range: 0.6077. Min Range: -0.6077

Max Act: 0.3991. Min Act: -0.0000

Data Index: 1416720 (Python Code)

Max Activating Token Index: 1011

Text #12

Max Range: 0.6077. Min Range: -0.6077

Max Act: 0.4195. Min Act: -0.0000

Data Index: 1482252 (Python Code)

Max Activating Token Index: 38

Text #13

Max Range: 0.6077. Min Range: -0.6077

Max Act: 0.3989. Min Act: -0.0000

Data Index: 1427360 (Python Code)

Max Activating Token Index: 206

Text #14

Max Range: 0.6077. Min Range: -0.6077

Max Act: 0.4298. Min Act: -0.0000

Data Index: 1389268 (Python Code)

Max Activating Token Index: 594

Text #15

Max Range: 0.6077. Min Range: -0.6077

Max Act: 0.4079. Min Act: -0.0000

Data Index: 1621217 (Python Code)

Max Activating Token Index: 645

Text #16

Max Range: 0.6077. Min Range: -0.6077

Max Act: 0.3911. Min Act: -0.0000

Data Index: 1390157 (Python Code)

Max Activating Token Index: 291

Text #17

Max Range: 0.6077. Min Range: -0.6077

Max Act: 0.3884. Min Act: -0.0000

Data Index: 1645761 (Python Code)

Max Activating Token Index: 865

Text #18

Max Range: 0.6077. Min Range: -0.6077

Max Act: 0.3889. Min Act: -0.0000

Data Index: 1544070 (Python Code)

Max Activating Token Index: 567

Text #19

Max Range: 0.6077. Min Range: -0.6077

Max Act: 0.4145. Min Act: -0.0000

Data Index: 1448382 (Python Code)

Max Activating Token Index: 906