Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4149 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.1125. Min Range: -1.1125

Max Act: 1.1125. Min Act: -0.0000

Data Index: 1524443 (Python Code)

Max Activating Token Index: 670

Text #1

Max Range: 1.1125. Min Range: -1.1125

Max Act: 0.9900. Min Act: -0.0000

Data Index: 1394340 (Python Code)

Max Activating Token Index: 117

Text #2

Max Range: 1.1125. Min Range: -1.1125

Max Act: 0.8352. Min Act: -0.0000

Data Index: 1377124 (Python Code)

Max Activating Token Index: 108

Text #3

Max Range: 1.1125. Min Range: -1.1125

Max Act: 0.7884. Min Act: -0.0000

Data Index: 1475406 (Python Code)

Max Activating Token Index: 498

Text #4

Max Range: 1.1125. Min Range: -1.1125

Max Act: 0.7946. Min Act: -0.0000

Data Index: 1537259 (Python Code)

Max Activating Token Index: 238

Text #5

Max Range: 1.1125. Min Range: -1.1125

Max Act: 0.7530. Min Act: 0.0000

Data Index: 1445679 (Python Code)

Max Activating Token Index: 293

Text #6

Max Range: 1.1125. Min Range: -1.1125

Max Act: 0.7133. Min Act: -0.0000

Data Index: 1581095 (Python Code)

Max Activating Token Index: 68

Text #7

Max Range: 1.1125. Min Range: -1.1125

Max Act: 0.7212. Min Act: -0.0000

Data Index: 1620776 (Python Code)

Max Activating Token Index: 841

Text #8

Max Range: 1.1125. Min Range: -1.1125

Max Act: 0.7094. Min Act: -0.0000

Data Index: 407601 (C4 (Web Text))

Max Activating Token Index: 36

Text #9

Max Range: 1.1125. Min Range: -1.1125

Max Act: 0.6589. Min Act: -0.0000

Data Index: 334800 (C4 (Web Text))

Max Activating Token Index: 724

Text #10

Max Range: 1.1125. Min Range: -1.1125

Max Act: 0.6858. Min Act: 0.0000

Data Index: 1377359 (Python Code)

Max Activating Token Index: 17

Text #11

Max Range: 1.1125. Min Range: -1.1125

Max Act: 0.6620. Min Act: -0.0000

Data Index: 1444473 (Python Code)

Max Activating Token Index: 82

Text #12

Max Range: 1.1125. Min Range: -1.1125

Max Act: 0.6348. Min Act: -0.0000

Data Index: 1641692 (Python Code)

Max Activating Token Index: 402

Text #13

Max Range: 1.1125. Min Range: -1.1125

Max Act: 0.6112. Min Act: -0.0000

Data Index: 1126760 (C4 (Web Text))

Max Activating Token Index: 391

Text #14

Max Range: 1.1125. Min Range: -1.1125

Max Act: 0.5646. Min Act: -0.0000

Data Index: 557822 (C4 (Web Text))

Max Activating Token Index: 832

Text #15

Max Range: 1.1125. Min Range: -1.1125

Max Act: 0.5805. Min Act: -0.0000

Data Index: 1583795 (Python Code)

Max Activating Token Index: 187

Text #16

Max Range: 1.1125. Min Range: -1.1125

Max Act: 0.5888. Min Act: -0.0000

Data Index: 564535 (C4 (Web Text))

Max Activating Token Index: 680

Text #17

Max Range: 1.1125. Min Range: -1.1125

Max Act: 0.6007. Min Act: -0.0000

Data Index: 1475655 (Python Code)

Max Activating Token Index: 945

Text #18

Max Range: 1.1125. Min Range: -1.1125

Max Act: 0.5543. Min Act: -0.0000

Data Index: 1405960 (Python Code)

Max Activating Token Index: 430

Text #19

Max Range: 1.1125. Min Range: -1.1125

Max Act: 0.5297. Min Act: -0.0000

Data Index: 1631652 (Python Code)

Max Activating Token Index: 533