Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4223 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.1382. Min Range: -1.1382

Max Act: 1.0320. Min Act: -0.0000

Data Index: 1437088 (Python Code)

Max Activating Token Index: 216

Text #1

Max Range: 1.1382. Min Range: -1.1382

Max Act: 0.9556. Min Act: -0.0000

Data Index: 1595085 (Python Code)

Max Activating Token Index: 714

Text #2

Max Range: 1.1382. Min Range: -1.1382

Max Act: 0.8432. Min Act: -0.0000

Data Index: 1451924 (Python Code)

Max Activating Token Index: 321

Text #3

Max Range: 1.1382. Min Range: -1.1382

Max Act: 1.1382. Min Act: -0.0000

Data Index: 1510959 (Python Code)

Max Activating Token Index: 821

Text #4

Max Range: 1.1382. Min Range: -1.1382

Max Act: 0.8312. Min Act: -0.0000

Data Index: 1421446 (Python Code)

Max Activating Token Index: 614

Text #5

Max Range: 1.1382. Min Range: -1.1382

Max Act: 0.8722. Min Act: -0.0000

Data Index: 1631633 (Python Code)

Max Activating Token Index: 266

Text #6

Max Range: 1.1382. Min Range: -1.1382

Max Act: 0.7890. Min Act: -0.0000

Data Index: 1513850 (Python Code)

Max Activating Token Index: 695

Text #7

Max Range: 1.1382. Min Range: -1.1382

Max Act: 0.8826. Min Act: -0.0000

Data Index: 1523259 (Python Code)

Max Activating Token Index: 367

Text #8

Max Range: 1.1382. Min Range: -1.1382

Max Act: 0.9145. Min Act: -0.0000

Data Index: 1648617 (Python Code)

Max Activating Token Index: 93

Text #9

Max Range: 1.1382. Min Range: -1.1382

Max Act: 0.8903. Min Act: -0.0000

Data Index: 1500108 (Python Code)

Max Activating Token Index: 915

Text #10

Max Range: 1.1382. Min Range: -1.1382

Max Act: 0.8317. Min Act: -0.0000

Data Index: 1542186 (Python Code)

Max Activating Token Index: 689

Text #11

Max Range: 1.1382. Min Range: -1.1382

Max Act: 0.9051. Min Act: -0.0000

Data Index: 1633618 (Python Code)

Max Activating Token Index: 84

Text #12

Max Range: 1.1382. Min Range: -1.1382

Max Act: 0.8751. Min Act: -0.0000

Data Index: 1498924 (Python Code)

Max Activating Token Index: 166

Text #13

Max Range: 1.1382. Min Range: -1.1382

Max Act: 0.7284. Min Act: -0.0000

Data Index: 1519322 (Python Code)

Max Activating Token Index: 264

Text #14

Max Range: 1.1382. Min Range: -1.1382

Max Act: 0.8506. Min Act: -0.0000

Data Index: 1557815 (Python Code)

Max Activating Token Index: 90

Text #15

Max Range: 1.1382. Min Range: -1.1382

Max Act: 0.9528. Min Act: -0.0000

Data Index: 1560266 (Python Code)

Max Activating Token Index: 280

Text #16

Max Range: 1.1382. Min Range: -1.1382

Max Act: 0.7535. Min Act: -0.0000

Data Index: 1577665 (Python Code)

Max Activating Token Index: 869

Text #17

Max Range: 1.1382. Min Range: -1.1382

Max Act: 0.9166. Min Act: -0.0000

Data Index: 1570739 (Python Code)

Max Activating Token Index: 701

Text #18

Max Range: 1.1382. Min Range: -1.1382

Max Act: 0.8284. Min Act: -0.0000

Data Index: 1646544 (Python Code)

Max Activating Token Index: 163

Text #19

Max Range: 1.1382. Min Range: -1.1382

Max Act: 0.7893. Min Act: -0.0000

Data Index: 1624932 (Python Code)

Max Activating Token Index: 792