Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1866 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 0.2230. Min Range: -0.2230

Max Act: 0.2230. Min Act: -0.0000

Data Index: 394715 (C4 (Web Text))

Max Activating Token Index: 220

Text #1

Max Range: 0.2230. Min Range: -0.2230

Max Act: 0.2135. Min Act: -0.0000

Data Index: 760299 (C4 (Web Text))

Max Activating Token Index: 922

Text #2

Max Range: 0.2230. Min Range: -0.2230

Max Act: 0.1787. Min Act: -0.0000

Data Index: 421147 (C4 (Web Text))

Max Activating Token Index: 713

Text #3

Max Range: 0.2230. Min Range: -0.2230

Max Act: 0.1820. Min Act: -0.0000

Data Index: 655351 (C4 (Web Text))

Max Activating Token Index: 470

Text #4

Max Range: 0.2230. Min Range: -0.2230

Max Act: 0.1768. Min Act: -0.0000

Data Index: 725106 (C4 (Web Text))

Max Activating Token Index: 379

Text #5

Max Range: 0.2230. Min Range: -0.2230

Max Act: 0.1863. Min Act: -0.0000

Data Index: 895503 (C4 (Web Text))

Max Activating Token Index: 210

Text #6

Max Range: 0.2230. Min Range: -0.2230

Max Act: 0.1766. Min Act: -0.0000

Data Index: 1362869 (Python Code)

Max Activating Token Index: 397

Text #7

Max Range: 0.2230. Min Range: -0.2230

Max Act: 0.1621. Min Act: -0.0000

Data Index: 1236605 (C4 (Web Text))

Max Activating Token Index: 78

Text #8

Max Range: 0.2230. Min Range: -0.2230

Max Act: 0.1614. Min Act: -0.0000

Data Index: 720943 (C4 (Web Text))

Max Activating Token Index: 2

Text #9

Max Range: 0.2230. Min Range: -0.2230

Max Act: 0.1701. Min Act: -0.0000

Data Index: 1198844 (C4 (Web Text))

Max Activating Token Index: 34

Text #10

Max Range: 0.2230. Min Range: -0.2230

Max Act: 0.1657. Min Act: -0.0000

Data Index: 378781 (C4 (Web Text))

Max Activating Token Index: 21

Text #11

Max Range: 0.2230. Min Range: -0.2230

Max Act: 0.1654. Min Act: -0.0000

Data Index: 279361 (C4 (Web Text))

Max Activating Token Index: 617

Text #12

Max Range: 0.2230. Min Range: -0.2230

Max Act: 0.1546. Min Act: -0.0000

Data Index: 102452 (C4 (Web Text))

Max Activating Token Index: 650

Text #13

Max Range: 0.2230. Min Range: -0.2230

Max Act: 0.1501. Min Act: -0.0000

Data Index: 712816 (C4 (Web Text))

Max Activating Token Index: 244

Text #14

Max Range: 0.2230. Min Range: -0.2230

Max Act: 0.1446. Min Act: -0.0000

Data Index: 281478 (C4 (Web Text))

Max Activating Token Index: 551

Text #15

Max Range: 0.2230. Min Range: -0.2230

Max Act: 0.1592. Min Act: -0.0000

Data Index: 361422 (C4 (Web Text))

Max Activating Token Index: 446

Text #16

Max Range: 0.2230. Min Range: -0.2230

Max Act: 0.1557. Min Act: -0.0000

Data Index: 581652 (C4 (Web Text))

Max Activating Token Index: 514

Text #17

Max Range: 0.2230. Min Range: -0.2230

Max Act: 0.1461. Min Act: -0.0000

Data Index: 899818 (C4 (Web Text))

Max Activating Token Index: 985

Text #18

Max Range: 0.2230. Min Range: -0.2230

Max Act: 0.1530. Min Act: -0.0000

Data Index: 677870 (C4 (Web Text))

Max Activating Token Index: 29

Text #19

Max Range: 0.2230. Min Range: -0.2230

Max Act: 0.1506. Min Act: -0.0000

Data Index: 398792 (C4 (Web Text))

Max Activating Token Index: 270