Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1285 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 3.8398. Min Range: -3.8398

Max Act: 3.7091. Min Act: -0.0000

Data Index: 1540364 (Python Code)

Max Activating Token Index: 632

Text #1

Max Range: 3.8398. Min Range: -3.8398

Max Act: 3.8398. Min Act: -0.0000

Data Index: 1367267 (Python Code)

Max Activating Token Index: 229

Text #2

Max Range: 3.8398. Min Range: -3.8398

Max Act: 3.2844. Min Act: -0.0000

Data Index: 1501584 (Python Code)

Max Activating Token Index: 712

Text #3

Max Range: 3.8398. Min Range: -3.8398

Max Act: 3.4912. Min Act: -0.0000

Data Index: 1638089 (Python Code)

Max Activating Token Index: 760

Text #4

Max Range: 3.8398. Min Range: -3.8398

Max Act: 3.1999. Min Act: -0.0000

Data Index: 1495869 (Python Code)

Max Activating Token Index: 539

Text #5

Max Range: 3.8398. Min Range: -3.8398

Max Act: 2.9671. Min Act: 0.0000

Data Index: 1390233 (Python Code)

Max Activating Token Index: 860

Text #6

Max Range: 3.8398. Min Range: -3.8398

Max Act: 2.7747. Min Act: -0.0000

Data Index: 1515376 (Python Code)

Max Activating Token Index: 431

Text #7

Max Range: 3.8398. Min Range: -3.8398

Max Act: 2.7699. Min Act: 0.0000

Data Index: 1457481 (Python Code)

Max Activating Token Index: 745

Text #8

Max Range: 3.8398. Min Range: -3.8398

Max Act: 2.8907. Min Act: -0.0000

Data Index: 1367485 (Python Code)

Max Activating Token Index: 612

Text #9

Max Range: 3.8398. Min Range: -3.8398

Max Act: 2.6652. Min Act: -0.0000

Data Index: 1635550 (Python Code)

Max Activating Token Index: 355

Text #10

Max Range: 3.8398. Min Range: -3.8398

Max Act: 2.5078. Min Act: 0.0000

Data Index: 1384376 (Python Code)

Max Activating Token Index: 759

Text #11

Max Range: 3.8398. Min Range: -3.8398

Max Act: 2.6353. Min Act: -0.0000

Data Index: 1492517 (Python Code)

Max Activating Token Index: 660

Text #12

Max Range: 3.8398. Min Range: -3.8398

Max Act: 2.4071. Min Act: -0.0000

Data Index: 1373520 (Python Code)

Max Activating Token Index: 229

Text #13

Max Range: 3.8398. Min Range: -3.8398

Max Act: 2.6630. Min Act: 0.0001

Data Index: 1387755 (Python Code)

Max Activating Token Index: 193

Text #14

Max Range: 3.8398. Min Range: -3.8398

Max Act: 2.4993. Min Act: 0.0000

Data Index: 1450854 (Python Code)

Max Activating Token Index: 65

Text #15

Max Range: 3.8398. Min Range: -3.8398

Max Act: 2.6451. Min Act: -0.0000

Data Index: 1454285 (Python Code)

Max Activating Token Index: 1023

Text #16

Max Range: 3.8398. Min Range: -3.8398

Max Act: 2.3907. Min Act: -0.0000

Data Index: 1628630 (Python Code)

Max Activating Token Index: 224

Text #17

Max Range: 3.8398. Min Range: -3.8398

Max Act: 2.5036. Min Act: -0.0000

Data Index: 1531949 (Python Code)

Max Activating Token Index: 333

Text #18

Max Range: 3.8398. Min Range: -3.8398

Max Act: 2.3707. Min Act: 0.0000

Data Index: 1634131 (Python Code)

Max Activating Token Index: 885

Text #19

Max Range: 3.8398. Min Range: -3.8398

Max Act: 2.4677. Min Act: -0.0000

Data Index: 1604154 (Python Code)

Max Activating Token Index: 158