Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 297 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.1520. Min Range: -1.1520

Max Act: 1.1520. Min Act: -0.0000

Data Index: 1374587 (Python Code)

Max Activating Token Index: 926

Text #1

Max Range: 1.1520. Min Range: -1.1520

Max Act: 0.9425. Min Act: -0.0000

Data Index: 1619303 (Python Code)

Max Activating Token Index: 760

Text #2

Max Range: 1.1520. Min Range: -1.1520

Max Act: 0.8831. Min Act: -0.0000

Data Index: 1576718 (Python Code)

Max Activating Token Index: 483

Text #3

Max Range: 1.1520. Min Range: -1.1520

Max Act: 0.7308. Min Act: -0.0000

Data Index: 1450827 (Python Code)

Max Activating Token Index: 574

Text #4

Max Range: 1.1520. Min Range: -1.1520

Max Act: 0.8229. Min Act: -0.0000

Data Index: 1606190 (Python Code)

Max Activating Token Index: 213

Text #5

Max Range: 1.1520. Min Range: -1.1520

Max Act: 0.8178. Min Act: -0.0000

Data Index: 1532474 (Python Code)

Max Activating Token Index: 1000

Text #6

Max Range: 1.1520. Min Range: -1.1520

Max Act: 0.7251. Min Act: -0.0000

Data Index: 1382583 (Python Code)

Max Activating Token Index: 175

Text #7

Max Range: 1.1520. Min Range: -1.1520

Max Act: 0.8051. Min Act: -0.0000

Data Index: 1654707 (Python Code)

Max Activating Token Index: 1021

Text #8

Max Range: 1.1520. Min Range: -1.1520

Max Act: 0.7276. Min Act: -0.0000

Data Index: 1579985 (Python Code)

Max Activating Token Index: 668

Text #9

Max Range: 1.1520. Min Range: -1.1520

Max Act: 0.7328. Min Act: -0.0000

Data Index: 1489766 (Python Code)

Max Activating Token Index: 649

Text #10

Max Range: 1.1520. Min Range: -1.1520

Max Act: 0.7311. Min Act: -0.0000

Data Index: 104114 (C4 (Web Text))

Max Activating Token Index: 579

Text #11

Max Range: 1.1520. Min Range: -1.1520

Max Act: 0.7179. Min Act: -0.0000

Data Index: 1458142 (Python Code)

Max Activating Token Index: 602

Text #12

Max Range: 1.1520. Min Range: -1.1520

Max Act: 0.8907. Min Act: -0.0000

Data Index: 1459149 (Python Code)

Max Activating Token Index: 569

Text #13

Max Range: 1.1520. Min Range: -1.1520

Max Act: 0.7788. Min Act: -0.0000

Data Index: 1496019 (Python Code)

Max Activating Token Index: 380

Text #14

Max Range: 1.1520. Min Range: -1.1520

Max Act: 0.7407. Min Act: -0.0000

Data Index: 1530108 (Python Code)

Max Activating Token Index: 512

Text #15

Max Range: 1.1520. Min Range: -1.1520

Max Act: 0.7475. Min Act: -0.0000

Data Index: 1420537 (Python Code)

Max Activating Token Index: 905

Text #16

Max Range: 1.1520. Min Range: -1.1520

Max Act: 0.7166. Min Act: -0.0000

Data Index: 1443130 (Python Code)

Max Activating Token Index: 359

Text #17

Max Range: 1.1520. Min Range: -1.1520

Max Act: 0.6744. Min Act: -0.0000

Data Index: 1585575 (Python Code)

Max Activating Token Index: 355

Text #18

Max Range: 1.1520. Min Range: -1.1520

Max Act: 0.7618. Min Act: -0.0000

Data Index: 1570886 (Python Code)

Max Activating Token Index: 912

Text #19

Max Range: 1.1520. Min Range: -1.1520

Max Act: 0.7505. Min Act: -0.0000

Data Index: 236069 (C4 (Web Text))

Max Activating Token Index: 268