Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1193 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.9978. Min Range: -0.9978

Max Act: 0.9965. Min Act: -0.0000

Data Index: 1635943 (Python Code)

Max Activating Token Index: 737

Text #1

Max Range: 0.9978. Min Range: -0.9978

Max Act: 0.9978. Min Act: -0.0000

Data Index: 1609388 (Python Code)

Max Activating Token Index: 643

Text #2

Max Range: 0.9978. Min Range: -0.9978

Max Act: 0.9698. Min Act: -0.0000

Data Index: 1498022 (Python Code)

Max Activating Token Index: 679

Text #3

Max Range: 0.9978. Min Range: -0.9978

Max Act: 0.9624. Min Act: -0.0000

Data Index: 1367432 (Python Code)

Max Activating Token Index: 544

Text #4

Max Range: 0.9978. Min Range: -0.9978

Max Act: 0.9519. Min Act: -0.0000

Data Index: 1604879 (Python Code)

Max Activating Token Index: 393

Text #5

Max Range: 0.9978. Min Range: -0.9978

Max Act: 0.9267. Min Act: -0.0000

Data Index: 1628585 (Python Code)

Max Activating Token Index: 60

Text #6

Max Range: 0.9978. Min Range: -0.9978

Max Act: 0.8896. Min Act: -0.0000

Data Index: 1476326 (Python Code)

Max Activating Token Index: 106

Text #7

Max Range: 0.9978. Min Range: -0.9978

Max Act: 0.9036. Min Act: -0.0000

Data Index: 1471737 (Python Code)

Max Activating Token Index: 138

Text #8

Max Range: 0.9978. Min Range: -0.9978

Max Act: 0.8347. Min Act: -0.0000

Data Index: 570822 (C4 (Web Text))

Max Activating Token Index: 309

Text #9

Max Range: 0.9978. Min Range: -0.9978

Max Act: 0.8061. Min Act: -0.0000

Data Index: 1485786 (Python Code)

Max Activating Token Index: 383

Text #10

Max Range: 0.9978. Min Range: -0.9978

Max Act: 0.8049. Min Act: -0.0000

Data Index: 1326598 (C4 (Web Text))

Max Activating Token Index: 264

Text #11

Max Range: 0.9978. Min Range: -0.9978

Max Act: 0.7995. Min Act: -0.0000

Data Index: 178398 (C4 (Web Text))

Max Activating Token Index: 551

Text #12

Max Range: 0.9978. Min Range: -0.9978

Max Act: 0.8208. Min Act: -0.0000

Data Index: 296767 (C4 (Web Text))

Max Activating Token Index: 746

Text #13

Max Range: 0.9978. Min Range: -0.9978

Max Act: 0.8215. Min Act: -0.0000

Data Index: 1451958 (Python Code)

Max Activating Token Index: 518

Text #14

Max Range: 0.9978. Min Range: -0.9978

Max Act: 0.8203. Min Act: -0.0000

Data Index: 656332 (C4 (Web Text))

Max Activating Token Index: 546

Text #15

Max Range: 0.9978. Min Range: -0.9978

Max Act: 0.7900. Min Act: -0.0000

Data Index: 1475184 (Python Code)

Max Activating Token Index: 781

Text #16

Max Range: 0.9978. Min Range: -0.9978

Max Act: 0.7819. Min Act: -0.0000

Data Index: 1475126 (Python Code)

Max Activating Token Index: 297

Text #17

Max Range: 0.9978. Min Range: -0.9978

Max Act: 0.7790. Min Act: -0.0000

Data Index: 1482593 (Python Code)

Max Activating Token Index: 606

Text #18

Max Range: 0.9978. Min Range: -0.9978

Max Act: 0.7526. Min Act: -0.0000

Data Index: 1474324 (Python Code)

Max Activating Token Index: 321

Text #19

Max Range: 0.9978. Min Range: -0.9978

Max Act: 0.8274. Min Act: -0.0000

Data Index: 659847 (C4 (Web Text))

Max Activating Token Index: 166