Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1607 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 4.7593. Min Range: -4.7593

Max Act: 4.6473. Min Act: -0.0000

Data Index: 290763 (C4 (Web Text))

Max Activating Token Index: 196

Text #1

Max Range: 4.7593. Min Range: -4.7593

Max Act: 4.7593. Min Act: -0.0000

Data Index: 1222390 (C4 (Web Text))

Max Activating Token Index: 817

Text #2

Max Range: 4.7593. Min Range: -4.7593

Max Act: 4.4606. Min Act: -0.0000

Data Index: 487451 (C4 (Web Text))

Max Activating Token Index: 235

Text #3

Max Range: 4.7593. Min Range: -4.7593

Max Act: 4.4219. Min Act: -0.0000

Data Index: 615222 (C4 (Web Text))

Max Activating Token Index: 439

Text #4

Max Range: 4.7593. Min Range: -4.7593

Max Act: 4.2114. Min Act: -0.0000

Data Index: 325894 (C4 (Web Text))

Max Activating Token Index: 840

Text #5

Max Range: 4.7593. Min Range: -4.7593

Max Act: 4.0224. Min Act: -0.0000

Data Index: 592811 (C4 (Web Text))

Max Activating Token Index: 450

Text #6

Max Range: 4.7593. Min Range: -4.7593

Max Act: 4.3038. Min Act: -0.0000

Data Index: 326453 (C4 (Web Text))

Max Activating Token Index: 633

Text #7

Max Range: 4.7593. Min Range: -4.7593

Max Act: 4.4090. Min Act: -0.0000

Data Index: 1069320 (C4 (Web Text))

Max Activating Token Index: 199

Text #8

Max Range: 4.7593. Min Range: -4.7593

Max Act: 4.0388. Min Act: 0.0000

Data Index: 569875 (C4 (Web Text))

Max Activating Token Index: 19

Text #9

Max Range: 4.7593. Min Range: -4.7593

Max Act: 3.8370. Min Act: -0.0000

Data Index: 827591 (C4 (Web Text))

Max Activating Token Index: 747

Text #10

Max Range: 4.7593. Min Range: -4.7593

Max Act: 3.8308. Min Act: -0.0000

Data Index: 796601 (C4 (Web Text))

Max Activating Token Index: 499

Text #11

Max Range: 4.7593. Min Range: -4.7593

Max Act: 3.6557. Min Act: -0.0000

Data Index: 338581 (C4 (Web Text))

Max Activating Token Index: 295

Text #12

Max Range: 4.7593. Min Range: -4.7593

Max Act: 4.1056. Min Act: -0.0000

Data Index: 838994 (C4 (Web Text))

Max Activating Token Index: 497

Text #13

Max Range: 4.7593. Min Range: -4.7593

Max Act: 3.9065. Min Act: -0.0000

Data Index: 396794 (C4 (Web Text))

Max Activating Token Index: 177

Text #14

Max Range: 4.7593. Min Range: -4.7593

Max Act: 3.6795. Min Act: -0.0000

Data Index: 1296995 (C4 (Web Text))

Max Activating Token Index: 148

Text #15

Max Range: 4.7593. Min Range: -4.7593

Max Act: 3.8311. Min Act: -0.0000

Data Index: 86132 (C4 (Web Text))

Max Activating Token Index: 284

Text #16

Max Range: 4.7593. Min Range: -4.7593

Max Act: 3.7848. Min Act: -0.0000

Data Index: 958765 (C4 (Web Text))

Max Activating Token Index: 26

Text #17

Max Range: 4.7593. Min Range: -4.7593

Max Act: 3.8667. Min Act: -0.0000

Data Index: 537875 (C4 (Web Text))

Max Activating Token Index: 543

Text #18

Max Range: 4.7593. Min Range: -4.7593

Max Act: 3.3446. Min Act: -0.0000

Data Index: 719716 (C4 (Web Text))

Max Activating Token Index: 93

Text #19

Max Range: 4.7593. Min Range: -4.7593

Max Act: 3.5085. Min Act: -0.0000

Data Index: 47082 (C4 (Web Text))

Max Activating Token Index: 586