Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 694 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.5326. Min Range: -1.5326

Max Act: 1.5326. Min Act: -0.0000

Data Index: 1194863 (C4 (Web Text))

Max Activating Token Index: 924

Text #1

Max Range: 1.5326. Min Range: -1.5326

Max Act: 1.4814. Min Act: -0.0000

Data Index: 457271 (C4 (Web Text))

Max Activating Token Index: 819

Text #2

Max Range: 1.5326. Min Range: -1.5326

Max Act: 1.3661. Min Act: -0.0000

Data Index: 24171 (C4 (Web Text))

Max Activating Token Index: 570

Text #3

Max Range: 1.5326. Min Range: -1.5326

Max Act: 1.4327. Min Act: -0.0000

Data Index: 1114166 (C4 (Web Text))

Max Activating Token Index: 128

Text #4

Max Range: 1.5326. Min Range: -1.5326

Max Act: 1.2732. Min Act: -0.0000

Data Index: 715653 (C4 (Web Text))

Max Activating Token Index: 118

Text #5

Max Range: 1.5326. Min Range: -1.5326

Max Act: 1.3240. Min Act: -0.0000

Data Index: 1150532 (C4 (Web Text))

Max Activating Token Index: 295

Text #6

Max Range: 1.5326. Min Range: -1.5326

Max Act: 1.2799. Min Act: -0.0000

Data Index: 754581 (C4 (Web Text))

Max Activating Token Index: 936

Text #7

Max Range: 1.5326. Min Range: -1.5326

Max Act: 1.3125. Min Act: -0.0000

Data Index: 554762 (C4 (Web Text))

Max Activating Token Index: 182

Text #8

Max Range: 1.5326. Min Range: -1.5326

Max Act: 1.3004. Min Act: -0.0000

Data Index: 1282451 (C4 (Web Text))

Max Activating Token Index: 740

Text #9

Max Range: 1.5326. Min Range: -1.5326

Max Act: 1.2328. Min Act: -0.0000

Data Index: 392273 (C4 (Web Text))

Max Activating Token Index: 254

Text #10

Max Range: 1.5326. Min Range: -1.5326

Max Act: 1.2506. Min Act: -0.0000

Data Index: 994688 (C4 (Web Text))

Max Activating Token Index: 198

Text #11

Max Range: 1.5326. Min Range: -1.5326

Max Act: 1.3113. Min Act: -0.0000

Data Index: 668687 (C4 (Web Text))

Max Activating Token Index: 272

Text #12

Max Range: 1.5326. Min Range: -1.5326

Max Act: 1.3072. Min Act: -0.0000

Data Index: 978879 (C4 (Web Text))

Max Activating Token Index: 362

Text #13

Max Range: 1.5326. Min Range: -1.5326

Max Act: 1.2940. Min Act: -0.0000

Data Index: 738747 (C4 (Web Text))

Max Activating Token Index: 479

Text #14

Max Range: 1.5326. Min Range: -1.5326

Max Act: 1.2238. Min Act: -0.0000

Data Index: 427997 (C4 (Web Text))

Max Activating Token Index: 284

Text #15

Max Range: 1.5326. Min Range: -1.5326

Max Act: 1.2420. Min Act: -0.0000

Data Index: 1267325 (C4 (Web Text))

Max Activating Token Index: 804

Text #16

Max Range: 1.5326. Min Range: -1.5326

Max Act: 1.2391. Min Act: -0.0000

Data Index: 247764 (C4 (Web Text))

Max Activating Token Index: 338

Text #17

Max Range: 1.5326. Min Range: -1.5326

Max Act: 1.1768. Min Act: -0.0000

Data Index: 668934 (C4 (Web Text))

Max Activating Token Index: 215

Text #18

Max Range: 1.5326. Min Range: -1.5326

Max Act: 1.2623. Min Act: -0.0000

Data Index: 158619 (C4 (Web Text))

Max Activating Token Index: 838

Text #19

Max Range: 1.5326. Min Range: -1.5326

Max Act: 1.1861. Min Act: -0.0000

Data Index: 1122199 (C4 (Web Text))

Max Activating Token Index: 582