Model: SoLU Model: 6 Layers, 3072 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 138 in Layer 3

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-6l')

Text #0

Max Range: 0.9503. Min Range: -0.9503

Max Act: 0.9503. Min Act: -0.0001

Data Index: 1099662 (C4 (Web Text))

Max Activating Token Index: 1008

Text #1

Max Range: 0.9503. Min Range: -0.9503

Max Act: 0.6942. Min Act: -0.0001

Data Index: 166722 (C4 (Web Text))

Max Activating Token Index: 263

Text #2

Max Range: 0.9503. Min Range: -0.9503

Max Act: 0.7834. Min Act: -0.0001

Data Index: 86127 (C4 (Web Text))

Max Activating Token Index: 112

Text #3

Max Range: 0.9503. Min Range: -0.9503

Max Act: 0.7149. Min Act: -0.0001

Data Index: 378057 (C4 (Web Text))

Max Activating Token Index: 276

Text #4

Max Range: 0.9503. Min Range: -0.9503

Max Act: 0.6500. Min Act: -0.0001

Data Index: 1140231 (C4 (Web Text))

Max Activating Token Index: 601

Text #5

Max Range: 0.9503. Min Range: -0.9503

Max Act: 0.7256. Min Act: -0.0001

Data Index: 637616 (C4 (Web Text))

Max Activating Token Index: 339

Text #6

Max Range: 0.9503. Min Range: -0.9503

Max Act: 0.6941. Min Act: -0.0001

Data Index: 982963 (C4 (Web Text))

Max Activating Token Index: 429

Text #7

Max Range: 0.9503. Min Range: -0.9503

Max Act: 0.6573. Min Act: -0.0001

Data Index: 43706 (C4 (Web Text))

Max Activating Token Index: 209

Text #8

Max Range: 0.9503. Min Range: -0.9503

Max Act: 0.6937. Min Act: -0.0001

Data Index: 967278 (C4 (Web Text))

Max Activating Token Index: 575

Text #9

Max Range: 0.9503. Min Range: -0.9503

Max Act: 0.6864. Min Act: -0.0001

Data Index: 1326955 (C4 (Web Text))

Max Activating Token Index: 973

Text #10

Max Range: 0.9503. Min Range: -0.9503

Max Act: 0.5405. Min Act: -0.0001

Data Index: 1164370 (C4 (Web Text))

Max Activating Token Index: 342

Text #11

Max Range: 0.9503. Min Range: -0.9503

Max Act: 0.5656. Min Act: -0.0001

Data Index: 619868 (C4 (Web Text))

Max Activating Token Index: 314

Text #12

Max Range: 0.9503. Min Range: -0.9503

Max Act: 0.6403. Min Act: -0.0001

Data Index: 709916 (C4 (Web Text))

Max Activating Token Index: 327

Text #13

Max Range: 0.9503. Min Range: -0.9503

Max Act: 0.6683. Min Act: -0.0001

Data Index: 78460 (C4 (Web Text))

Max Activating Token Index: 606

Text #14

Max Range: 0.9503. Min Range: -0.9503

Max Act: 0.6518. Min Act: -0.0001

Data Index: 375870 (C4 (Web Text))

Max Activating Token Index: 853

Text #15

Max Range: 0.9503. Min Range: -0.9503

Max Act: 0.6662. Min Act: -0.0001

Data Index: 851908 (C4 (Web Text))

Max Activating Token Index: 78

Text #16

Max Range: 0.9503. Min Range: -0.9503

Max Act: 0.6488. Min Act: -0.0001

Data Index: 192681 (C4 (Web Text))

Max Activating Token Index: 811

Text #17

Max Range: 0.9503. Min Range: -0.9503

Max Act: 0.6448. Min Act: -0.0001

Data Index: 48650 (C4 (Web Text))

Max Activating Token Index: 714

Text #18

Max Range: 0.9503. Min Range: -0.9503

Max Act: 0.6042. Min Act: -0.0001

Data Index: 1262196 (C4 (Web Text))

Max Activating Token Index: 548

Text #19

Max Range: 0.9503. Min Range: -0.9503

Max Act: 0.6193. Min Act: -0.0001

Data Index: 913122 (C4 (Web Text))

Max Activating Token Index: 1005