Model: SoLU Model: 8 Layers, 4096 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3522 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-8l')

Text #0

Max Range: 3.3701. Min Range: -3.3701

Max Act: 3.3701. Min Act: -0.0000

Data Index: 622151 (C4 (Web Text))

Max Activating Token Index: 371

Text #1

Max Range: 3.3701. Min Range: -3.3701

Max Act: 3.2073. Min Act: -0.0001

Data Index: 340486 (C4 (Web Text))

Max Activating Token Index: 66

Text #2

Max Range: 3.3701. Min Range: -3.3701

Max Act: 3.0979. Min Act: 0.0000

Data Index: 939264 (C4 (Web Text))

Max Activating Token Index: 104

Text #3

Max Range: 3.3701. Min Range: -3.3701

Max Act: 2.7443. Min Act: -0.0000

Data Index: 471149 (C4 (Web Text))

Max Activating Token Index: 674

Text #4

Max Range: 3.3701. Min Range: -3.3701

Max Act: 3.0022. Min Act: -0.0000

Data Index: 563906 (C4 (Web Text))

Max Activating Token Index: 801

Text #5

Max Range: 3.3701. Min Range: -3.3701

Max Act: 2.8448. Min Act: -0.0000

Data Index: 820915 (C4 (Web Text))

Max Activating Token Index: 663

Text #6

Max Range: 3.3701. Min Range: -3.3701

Max Act: 2.8913. Min Act: -0.0000

Data Index: 94537 (C4 (Web Text))

Max Activating Token Index: 640

Text #7

Max Range: 3.3701. Min Range: -3.3701

Max Act: 2.5867. Min Act: -0.0000

Data Index: 850218 (C4 (Web Text))

Max Activating Token Index: 602

Text #8

Max Range: 3.3701. Min Range: -3.3701

Max Act: 2.7840. Min Act: -0.0000

Data Index: 221246 (C4 (Web Text))

Max Activating Token Index: 197

Text #9

Max Range: 3.3701. Min Range: -3.3701

Max Act: 2.7732. Min Act: -0.0000

Data Index: 1011190 (C4 (Web Text))

Max Activating Token Index: 347

Text #10

Max Range: 3.3701. Min Range: -3.3701

Max Act: 2.7794. Min Act: 0.0000

Data Index: 1047391 (C4 (Web Text))

Max Activating Token Index: 90

Text #11

Max Range: 3.3701. Min Range: -3.3701

Max Act: 2.5060. Min Act: -0.0000

Data Index: 656353 (C4 (Web Text))

Max Activating Token Index: 113

Text #12

Max Range: 3.3701. Min Range: -3.3701

Max Act: 2.5120. Min Act: -0.0000

Data Index: 1260307 (C4 (Web Text))

Max Activating Token Index: 521

Text #13

Max Range: 3.3701. Min Range: -3.3701

Max Act: 2.6183. Min Act: -0.0001

Data Index: 487846 (C4 (Web Text))

Max Activating Token Index: 36

Text #14

Max Range: 3.3701. Min Range: -3.3701

Max Act: 2.6108. Min Act: 0.0000

Data Index: 1123843 (C4 (Web Text))

Max Activating Token Index: 679

Text #15

Max Range: 3.3701. Min Range: -3.3701

Max Act: 2.7281. Min Act: -0.0000

Data Index: 1011918 (C4 (Web Text))

Max Activating Token Index: 891

Text #16

Max Range: 3.3701. Min Range: -3.3701

Max Act: 2.7153. Min Act: 0.0000

Data Index: 770211 (C4 (Web Text))

Max Activating Token Index: 987

Text #17

Max Range: 3.3701. Min Range: -3.3701

Max Act: 2.4363. Min Act: -0.0000

Data Index: 342014 (C4 (Web Text))

Max Activating Token Index: 781

Text #18

Max Range: 3.3701. Min Range: -3.3701

Max Act: 2.5674. Min Act: 0.0000

Data Index: 526487 (C4 (Web Text))

Max Activating Token Index: 797

Text #19

Max Range: 3.3701. Min Range: -3.3701

Max Act: 2.5462. Min Act: -0.0000

Data Index: 722525 (C4 (Web Text))

Max Activating Token Index: 679