Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 939 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.8952. Min Range: -0.8952

Max Act: 0.8952. Min Act: -0.0000

Data Index: 1280517 (C4 (Web Text))

Max Activating Token Index: 691

Text #1

Max Range: 0.8952. Min Range: -0.8952

Max Act: 0.8282. Min Act: -0.0000

Data Index: 566005 (C4 (Web Text))

Max Activating Token Index: 842

Text #2

Max Range: 0.8952. Min Range: -0.8952

Max Act: 0.8328. Min Act: -0.0000

Data Index: 476949 (C4 (Web Text))

Max Activating Token Index: 251

Text #3

Max Range: 0.8952. Min Range: -0.8952

Max Act: 0.8001. Min Act: -0.0000

Data Index: 1103892 (C4 (Web Text))

Max Activating Token Index: 574

Text #4

Max Range: 0.8952. Min Range: -0.8952

Max Act: 0.7481. Min Act: -0.0000

Data Index: 602905 (C4 (Web Text))

Max Activating Token Index: 827

Text #5

Max Range: 0.8952. Min Range: -0.8952

Max Act: 0.7614. Min Act: -0.0000

Data Index: 399796 (C4 (Web Text))

Max Activating Token Index: 654

Text #6

Max Range: 0.8952. Min Range: -0.8952

Max Act: 0.7393. Min Act: -0.0000

Data Index: 1141193 (C4 (Web Text))

Max Activating Token Index: 390

Text #7

Max Range: 0.8952. Min Range: -0.8952

Max Act: 0.7286. Min Act: -0.0000

Data Index: 571485 (C4 (Web Text))

Max Activating Token Index: 501

Text #8

Max Range: 0.8952. Min Range: -0.8952

Max Act: 0.6737. Min Act: -0.0000

Data Index: 605276 (C4 (Web Text))

Max Activating Token Index: 344

Text #9

Max Range: 0.8952. Min Range: -0.8952

Max Act: 0.7295. Min Act: -0.0000

Data Index: 546729 (C4 (Web Text))

Max Activating Token Index: 684

Text #10

Max Range: 0.8952. Min Range: -0.8952

Max Act: 0.6762. Min Act: -0.0000

Data Index: 1210043 (C4 (Web Text))

Max Activating Token Index: 31

Text #11

Max Range: 0.8952. Min Range: -0.8952

Max Act: 0.7134. Min Act: -0.0000

Data Index: 136825 (C4 (Web Text))

Max Activating Token Index: 343

Text #12

Max Range: 0.8952. Min Range: -0.8952

Max Act: 0.6143. Min Act: -0.0000

Data Index: 260731 (C4 (Web Text))

Max Activating Token Index: 211

Text #13

Max Range: 0.8952. Min Range: -0.8952

Max Act: 0.6664. Min Act: -0.0000

Data Index: 890801 (C4 (Web Text))

Max Activating Token Index: 590

Text #14

Max Range: 0.8952. Min Range: -0.8952

Max Act: 0.6892. Min Act: -0.0000

Data Index: 1073687 (C4 (Web Text))

Max Activating Token Index: 52

Text #15

Max Range: 0.8952. Min Range: -0.8952

Max Act: 0.6260. Min Act: -0.0000

Data Index: 1211743 (C4 (Web Text))

Max Activating Token Index: 146

Text #16

Max Range: 0.8952. Min Range: -0.8952

Max Act: 0.6072. Min Act: -0.0000

Data Index: 419794 (C4 (Web Text))

Max Activating Token Index: 452

Text #17

Max Range: 0.8952. Min Range: -0.8952

Max Act: 0.6609. Min Act: -0.0000

Data Index: 108126 (C4 (Web Text))

Max Activating Token Index: 717

Text #18

Max Range: 0.8952. Min Range: -0.8952

Max Act: 0.7022. Min Act: -0.0000

Data Index: 355840 (C4 (Web Text))

Max Activating Token Index: 29

Text #19

Max Range: 0.8952. Min Range: -0.8952

Max Act: 0.6234. Min Act: -0.0000

Data Index: 753521 (C4 (Web Text))

Max Activating Token Index: 435