Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 551 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.9508. Min Range: -0.9508

Max Act: 0.9508. Min Act: -0.0000

Data Index: 874968 (C4 (Web Text))

Max Activating Token Index: 419

Text #1

Max Range: 0.9508. Min Range: -0.9508

Max Act: 0.7616. Min Act: -0.0000

Data Index: 158831 (C4 (Web Text))

Max Activating Token Index: 910

Text #2

Max Range: 0.9508. Min Range: -0.9508

Max Act: 0.8525. Min Act: -0.0000

Data Index: 358729 (C4 (Web Text))

Max Activating Token Index: 1001

Text #3

Max Range: 0.9508. Min Range: -0.9508

Max Act: 0.8551. Min Act: -0.0000

Data Index: 559708 (C4 (Web Text))

Max Activating Token Index: 822

Text #4

Max Range: 0.9508. Min Range: -0.9508

Max Act: 0.7971. Min Act: -0.0000

Data Index: 1078076 (C4 (Web Text))

Max Activating Token Index: 820

Text #5

Max Range: 0.9508. Min Range: -0.9508

Max Act: 0.8596. Min Act: -0.0000

Data Index: 1100090 (C4 (Web Text))

Max Activating Token Index: 971

Text #6

Max Range: 0.9508. Min Range: -0.9508

Max Act: 0.8020. Min Act: -0.0000

Data Index: 379593 (C4 (Web Text))

Max Activating Token Index: 178

Text #7

Max Range: 0.9508. Min Range: -0.9508

Max Act: 0.7176. Min Act: -0.0000

Data Index: 473758 (C4 (Web Text))

Max Activating Token Index: 965

Text #8

Max Range: 0.9508. Min Range: -0.9508

Max Act: 0.7063. Min Act: 0.0000

Data Index: 936414 (C4 (Web Text))

Max Activating Token Index: 958

Text #9

Max Range: 0.9508. Min Range: -0.9508

Max Act: 0.7209. Min Act: -0.0000

Data Index: 372267 (C4 (Web Text))

Max Activating Token Index: 65

Text #10

Max Range: 0.9508. Min Range: -0.9508

Max Act: 0.7622. Min Act: -0.0000

Data Index: 432833 (C4 (Web Text))

Max Activating Token Index: 297

Text #11

Max Range: 0.9508. Min Range: -0.9508

Max Act: 0.7065. Min Act: -0.0000

Data Index: 171775 (C4 (Web Text))

Max Activating Token Index: 42

Text #12

Max Range: 0.9508. Min Range: -0.9508

Max Act: 0.6666. Min Act: -0.0000

Data Index: 1463856 (Python Code)

Max Activating Token Index: 210

Text #13

Max Range: 0.9508. Min Range: -0.9508

Max Act: 0.6755. Min Act: -0.0000

Data Index: 485690 (C4 (Web Text))

Max Activating Token Index: 476

Text #14

Max Range: 0.9508. Min Range: -0.9508

Max Act: 0.6382. Min Act: -0.0000

Data Index: 1460714 (Python Code)

Max Activating Token Index: 309

Text #15

Max Range: 0.9508. Min Range: -0.9508

Max Act: 0.6583. Min Act: -0.0000

Data Index: 989775 (C4 (Web Text))

Max Activating Token Index: 127

Text #16

Max Range: 0.9508. Min Range: -0.9508

Max Act: 0.6363. Min Act: -0.0000

Data Index: 4291 (C4 (Web Text))

Max Activating Token Index: 364

Text #17

Max Range: 0.9508. Min Range: -0.9508

Max Act: 0.6554. Min Act: -0.0000

Data Index: 1070383 (C4 (Web Text))

Max Activating Token Index: 766

Text #18

Max Range: 0.9508. Min Range: -0.9508

Max Act: 0.6053. Min Act: 0.0001

Data Index: 1241195 (C4 (Web Text))

Max Activating Token Index: 57

Text #19

Max Range: 0.9508. Min Range: -0.9508

Max Act: 0.6144. Min Act: -0.0000

Data Index: 86071 (C4 (Web Text))

Max Activating Token Index: 333