Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3863 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.7558. Min Range: -0.7558

Max Act: 0.7162. Min Act: -0.0002

Data Index: 1130984 (C4 (Web Text))

Max Activating Token Index: 952

Text #1

Max Range: 0.7558. Min Range: -0.7558

Max Act: 0.7367. Min Act: -0.0002

Data Index: 400454 (C4 (Web Text))

Max Activating Token Index: 668

Text #2

Max Range: 0.7558. Min Range: -0.7558

Max Act: 0.7125. Min Act: -0.0002

Data Index: 1027326 (C4 (Web Text))

Max Activating Token Index: 989

Text #3

Max Range: 0.7558. Min Range: -0.7558

Max Act: 0.6880. Min Act: -0.0002

Data Index: 351021 (C4 (Web Text))

Max Activating Token Index: 1008

Text #4

Max Range: 0.7558. Min Range: -0.7558

Max Act: 0.5912. Min Act: -0.0002

Data Index: 205987 (C4 (Web Text))

Max Activating Token Index: 1020

Text #5

Max Range: 0.7558. Min Range: -0.7558

Max Act: 0.5691. Min Act: -0.0002

Data Index: 785741 (C4 (Web Text))

Max Activating Token Index: 843

Text #6

Max Range: 0.7558. Min Range: -0.7558

Max Act: 0.7558. Min Act: -0.0002

Data Index: 1194110 (C4 (Web Text))

Max Activating Token Index: 881

Text #7

Max Range: 0.7558. Min Range: -0.7558

Max Act: 0.6708. Min Act: -0.0002

Data Index: 901799 (C4 (Web Text))

Max Activating Token Index: 944

Text #8

Max Range: 0.7558. Min Range: -0.7558

Max Act: 0.6260. Min Act: -0.0002

Data Index: 63660 (C4 (Web Text))

Max Activating Token Index: 760

Text #9

Max Range: 0.7558. Min Range: -0.7558

Max Act: 0.6810. Min Act: -0.0002

Data Index: 944549 (C4 (Web Text))

Max Activating Token Index: 946

Text #10

Max Range: 0.7558. Min Range: -0.7558

Max Act: 0.6598. Min Act: -0.0002

Data Index: 201093 (C4 (Web Text))

Max Activating Token Index: 839

Text #11

Max Range: 0.7558. Min Range: -0.7558

Max Act: 0.6467. Min Act: -0.0002

Data Index: 575843 (C4 (Web Text))

Max Activating Token Index: 492

Text #12

Max Range: 0.7558. Min Range: -0.7558

Max Act: 0.6254. Min Act: -0.0002

Data Index: 611134 (C4 (Web Text))

Max Activating Token Index: 400

Text #13

Max Range: 0.7558. Min Range: -0.7558

Max Act: 0.6493. Min Act: -0.0002

Data Index: 714096 (C4 (Web Text))

Max Activating Token Index: 453

Text #14

Max Range: 0.7558. Min Range: -0.7558

Max Act: 0.6605. Min Act: -0.0002

Data Index: 364712 (C4 (Web Text))

Max Activating Token Index: 946

Text #15

Max Range: 0.7558. Min Range: -0.7558

Max Act: 0.5852. Min Act: -0.0002

Data Index: 663202 (C4 (Web Text))

Max Activating Token Index: 837

Text #16

Max Range: 0.7558. Min Range: -0.7558

Max Act: 0.5814. Min Act: -0.0002

Data Index: 1143130 (C4 (Web Text))

Max Activating Token Index: 992

Text #17

Max Range: 0.7558. Min Range: -0.7558

Max Act: 0.6322. Min Act: -0.0002

Data Index: 1143531 (C4 (Web Text))

Max Activating Token Index: 1014

Text #18

Max Range: 0.7558. Min Range: -0.7558

Max Act: 0.6551. Min Act: -0.0002

Data Index: 237268 (C4 (Web Text))

Max Activating Token Index: 162

Text #19

Max Range: 0.7558. Min Range: -0.7558

Max Act: 0.6544. Min Act: -0.0002

Data Index: 69400 (C4 (Web Text))

Max Activating Token Index: 457