Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2387 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.1283. Min Range: -1.1283

Max Act: 1.1283. Min Act: -0.0000

Data Index: 523880 (C4 (Web Text))

Max Activating Token Index: 464

Text #1

Max Range: 1.1283. Min Range: -1.1283

Max Act: 0.9857. Min Act: -0.0001

Data Index: 1070725 (C4 (Web Text))

Max Activating Token Index: 996

Text #2

Max Range: 1.1283. Min Range: -1.1283

Max Act: 0.9338. Min Act: -0.0000

Data Index: 154667 (C4 (Web Text))

Max Activating Token Index: 510

Text #3

Max Range: 1.1283. Min Range: -1.1283

Max Act: 0.9004. Min Act: -0.0000

Data Index: 1110858 (C4 (Web Text))

Max Activating Token Index: 27

Text #4

Max Range: 1.1283. Min Range: -1.1283

Max Act: 0.8978. Min Act: -0.0000

Data Index: 875527 (C4 (Web Text))

Max Activating Token Index: 154

Text #5

Max Range: 1.1283. Min Range: -1.1283

Max Act: 0.9676. Min Act: -0.0000

Data Index: 1081276 (C4 (Web Text))

Max Activating Token Index: 387

Text #6

Max Range: 1.1283. Min Range: -1.1283

Max Act: 0.8650. Min Act: -0.0000

Data Index: 274033 (C4 (Web Text))

Max Activating Token Index: 162

Text #7

Max Range: 1.1283. Min Range: -1.1283

Max Act: 0.8832. Min Act: -0.0000

Data Index: 766559 (C4 (Web Text))

Max Activating Token Index: 562

Text #8

Max Range: 1.1283. Min Range: -1.1283

Max Act: 0.8287. Min Act: -0.0000

Data Index: 382025 (C4 (Web Text))

Max Activating Token Index: 388

Text #9

Max Range: 1.1283. Min Range: -1.1283

Max Act: 0.8486. Min Act: -0.0000

Data Index: 666901 (C4 (Web Text))

Max Activating Token Index: 437

Text #10

Max Range: 1.1283. Min Range: -1.1283

Max Act: 0.7710. Min Act: -0.0000

Data Index: 1105641 (C4 (Web Text))

Max Activating Token Index: 591

Text #11

Max Range: 1.1283. Min Range: -1.1283

Max Act: 0.8798. Min Act: -0.0000

Data Index: 944927 (C4 (Web Text))

Max Activating Token Index: 568

Text #12

Max Range: 1.1283. Min Range: -1.1283

Max Act: 0.8066. Min Act: -0.0000

Data Index: 1146150 (C4 (Web Text))

Max Activating Token Index: 137

Text #13

Max Range: 1.1283. Min Range: -1.1283

Max Act: 0.7780. Min Act: -0.0000

Data Index: 54925 (C4 (Web Text))

Max Activating Token Index: 214

Text #14

Max Range: 1.1283. Min Range: -1.1283

Max Act: 0.7718. Min Act: -0.0001

Data Index: 335478 (C4 (Web Text))

Max Activating Token Index: 855

Text #15

Max Range: 1.1283. Min Range: -1.1283

Max Act: 0.7749. Min Act: -0.0000

Data Index: 1277564 (C4 (Web Text))

Max Activating Token Index: 720

Text #16

Max Range: 1.1283. Min Range: -1.1283

Max Act: 0.7283. Min Act: -0.0000

Data Index: 732610 (C4 (Web Text))

Max Activating Token Index: 405

Text #17

Max Range: 1.1283. Min Range: -1.1283

Max Act: 0.7621. Min Act: -0.0001

Data Index: 159329 (C4 (Web Text))

Max Activating Token Index: 265

Text #18

Max Range: 1.1283. Min Range: -1.1283

Max Act: 0.7353. Min Act: -0.0000

Data Index: 392663 (C4 (Web Text))

Max Activating Token Index: 33

Text #19

Max Range: 1.1283. Min Range: -1.1283

Max Act: 0.7438. Min Act: -0.0000

Data Index: 567160 (C4 (Web Text))

Max Activating Token Index: 23