Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3336 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.0903. Min Range: -1.0903

Max Act: 1.0903. Min Act: 0.0001

Data Index: 1426410 (Python Code)

Max Activating Token Index: 326

Text #1

Max Range: 1.0903. Min Range: -1.0903

Max Act: 1.0516. Min Act: -0.0000

Data Index: 1379351 (Python Code)

Max Activating Token Index: 910

Text #2

Max Range: 1.0903. Min Range: -1.0903

Max Act: 1.0591. Min Act: -0.0000

Data Index: 1504394 (Python Code)

Max Activating Token Index: 442

Text #3

Max Range: 1.0903. Min Range: -1.0903

Max Act: 1.0585. Min Act: 0.0001

Data Index: 1445317 (Python Code)

Max Activating Token Index: 142

Text #4

Max Range: 1.0903. Min Range: -1.0903

Max Act: 0.9606. Min Act: -0.0000

Data Index: 1544575 (Python Code)

Max Activating Token Index: 51

Text #5

Max Range: 1.0903. Min Range: -1.0903

Max Act: 0.9871. Min Act: 0.0002

Data Index: 1385273 (Python Code)

Max Activating Token Index: 237

Text #6

Max Range: 1.0903. Min Range: -1.0903

Max Act: 0.9286. Min Act: 0.0001

Data Index: 1459634 (Python Code)

Max Activating Token Index: 43

Text #7

Max Range: 1.0903. Min Range: -1.0903

Max Act: 0.9264. Min Act: 0.0001

Data Index: 1557043 (Python Code)

Max Activating Token Index: 358

Text #8

Max Range: 1.0903. Min Range: -1.0903

Max Act: 0.9514. Min Act: -0.0000

Data Index: 1504846 (Python Code)

Max Activating Token Index: 71

Text #9

Max Range: 1.0903. Min Range: -1.0903

Max Act: 0.8860. Min Act: 0.0003

Data Index: 1466272 (Python Code)

Max Activating Token Index: 153

Text #10

Max Range: 1.0903. Min Range: -1.0903

Max Act: 0.9198. Min Act: 0.0001

Data Index: 1488940 (Python Code)

Max Activating Token Index: 208

Text #11

Max Range: 1.0903. Min Range: -1.0903

Max Act: 0.8277. Min Act: -0.0000

Data Index: 1458139 (Python Code)

Max Activating Token Index: 901

Text #12

Max Range: 1.0903. Min Range: -1.0903

Max Act: 0.8124. Min Act: 0.0000

Data Index: 1360636 (Python Code)

Max Activating Token Index: 733

Text #13

Max Range: 1.0903. Min Range: -1.0903

Max Act: 0.8905. Min Act: 0.0000

Data Index: 1616225 (Python Code)

Max Activating Token Index: 21

Text #14

Max Range: 1.0903. Min Range: -1.0903

Max Act: 0.8264. Min Act: 0.0002

Data Index: 1636708 (Python Code)

Max Activating Token Index: 623

Text #15

Max Range: 1.0903. Min Range: -1.0903

Max Act: 0.8437. Min Act: 0.0003

Data Index: 1475879 (Python Code)

Max Activating Token Index: 386

Text #16

Max Range: 1.0903. Min Range: -1.0903

Max Act: 0.8092. Min Act: 0.0002

Data Index: 1425114 (Python Code)

Max Activating Token Index: 480

Text #17

Max Range: 1.0903. Min Range: -1.0903

Max Act: 0.8110. Min Act: 0.0000

Data Index: 1435895 (Python Code)

Max Activating Token Index: 452

Text #18

Max Range: 1.0903. Min Range: -1.0903

Max Act: 0.8074. Min Act: 0.0001

Data Index: 1626667 (Python Code)

Max Activating Token Index: 693

Text #19

Max Range: 1.0903. Min Range: -1.0903

Max Act: 0.7781. Min Act: 0.0000

Data Index: 1395947 (Python Code)

Max Activating Token Index: 118