Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1252 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.7440. Min Range: -0.7440

Max Act: 0.7438. Min Act: -0.0000

Data Index: 1501598 (Python Code)

Max Activating Token Index: 776

Text #1

Max Range: 0.7440. Min Range: -0.7440

Max Act: 0.7210. Min Act: -0.0000

Data Index: 1614335 (Python Code)

Max Activating Token Index: 518

Text #2

Max Range: 0.7440. Min Range: -0.7440

Max Act: 0.7440. Min Act: -0.0000

Data Index: 1640344 (Python Code)

Max Activating Token Index: 733

Text #3

Max Range: 0.7440. Min Range: -0.7440

Max Act: 0.6607. Min Act: -0.0000

Data Index: 1460195 (Python Code)

Max Activating Token Index: 341

Text #4

Max Range: 0.7440. Min Range: -0.7440

Max Act: 0.6902. Min Act: -0.0000

Data Index: 1556155 (Python Code)

Max Activating Token Index: 871

Text #5

Max Range: 0.7440. Min Range: -0.7440

Max Act: 0.6833. Min Act: -0.0000

Data Index: 1563349 (Python Code)

Max Activating Token Index: 815

Text #6

Max Range: 0.7440. Min Range: -0.7440

Max Act: 0.6799. Min Act: -0.0000

Data Index: 1556343 (Python Code)

Max Activating Token Index: 279

Text #7

Max Range: 0.7440. Min Range: -0.7440

Max Act: 0.6561. Min Act: -0.0000

Data Index: 1493386 (Python Code)

Max Activating Token Index: 953

Text #8

Max Range: 0.7440. Min Range: -0.7440

Max Act: 0.6089. Min Act: -0.0000

Data Index: 1437231 (Python Code)

Max Activating Token Index: 363

Text #9

Max Range: 0.7440. Min Range: -0.7440

Max Act: 0.6276. Min Act: -0.0000

Data Index: 1599671 (Python Code)

Max Activating Token Index: 865

Text #10

Max Range: 0.7440. Min Range: -0.7440

Max Act: 0.6116. Min Act: -0.0000

Data Index: 1413902 (Python Code)

Max Activating Token Index: 437

Text #11

Max Range: 0.7440. Min Range: -0.7440

Max Act: 0.6119. Min Act: -0.0000

Data Index: 1379602 (Python Code)

Max Activating Token Index: 899

Text #12

Max Range: 0.7440. Min Range: -0.7440

Max Act: 0.6291. Min Act: -0.0000

Data Index: 1577886 (Python Code)

Max Activating Token Index: 351

Text #13

Max Range: 0.7440. Min Range: -0.7440

Max Act: 0.5727. Min Act: -0.0000

Data Index: 1439654 (Python Code)

Max Activating Token Index: 732

Text #14

Max Range: 0.7440. Min Range: -0.7440

Max Act: 0.6102. Min Act: -0.0000

Data Index: 1588154 (Python Code)

Max Activating Token Index: 867

Text #15

Max Range: 0.7440. Min Range: -0.7440

Max Act: 0.6064. Min Act: -0.0000

Data Index: 1376131 (Python Code)

Max Activating Token Index: 373

Text #16

Max Range: 0.7440. Min Range: -0.7440

Max Act: 0.5774. Min Act: -0.0000

Data Index: 1394342 (Python Code)

Max Activating Token Index: 450

Text #17

Max Range: 0.7440. Min Range: -0.7440

Max Act: 0.5921. Min Act: -0.0000

Data Index: 1551154 (Python Code)

Max Activating Token Index: 594

Text #18

Max Range: 0.7440. Min Range: -0.7440

Max Act: 0.5945. Min Act: -0.0000

Data Index: 1601311 (Python Code)

Max Activating Token Index: 976

Text #19

Max Range: 0.7440. Min Range: -0.7440

Max Act: 0.5934. Min Act: -0.0000

Data Index: 1641238 (Python Code)

Max Activating Token Index: 1012