Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 866 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.3802. Min Range: -1.3802

Max Act: 1.3802. Min Act: -0.0000

Data Index: 147378 (C4 (Web Text))

Max Activating Token Index: 389

Text #1

Max Range: 1.3802. Min Range: -1.3802

Max Act: 1.3710. Min Act: -0.0000

Data Index: 1118311 (C4 (Web Text))

Max Activating Token Index: 150

Text #2

Max Range: 1.3802. Min Range: -1.3802

Max Act: 1.1517. Min Act: -0.0000

Data Index: 231822 (C4 (Web Text))

Max Activating Token Index: 58

Text #3

Max Range: 1.3802. Min Range: -1.3802

Max Act: 1.1581. Min Act: -0.0000

Data Index: 1191809 (C4 (Web Text))

Max Activating Token Index: 592

Text #4

Max Range: 1.3802. Min Range: -1.3802

Max Act: 1.0841. Min Act: -0.0000

Data Index: 544904 (C4 (Web Text))

Max Activating Token Index: 1001

Text #5

Max Range: 1.3802. Min Range: -1.3802

Max Act: 1.0874. Min Act: -0.0000

Data Index: 634454 (C4 (Web Text))

Max Activating Token Index: 282

Text #6

Max Range: 1.3802. Min Range: -1.3802

Max Act: 1.0525. Min Act: -0.0000

Data Index: 265277 (C4 (Web Text))

Max Activating Token Index: 695

Text #7

Max Range: 1.3802. Min Range: -1.3802

Max Act: 0.9801. Min Act: -0.0000

Data Index: 196411 (C4 (Web Text))

Max Activating Token Index: 463

Text #8

Max Range: 1.3802. Min Range: -1.3802

Max Act: 0.9966. Min Act: -0.0000

Data Index: 1197014 (C4 (Web Text))

Max Activating Token Index: 354

Text #9

Max Range: 1.3802. Min Range: -1.3802

Max Act: 0.9979. Min Act: -0.0000

Data Index: 569498 (C4 (Web Text))

Max Activating Token Index: 123

Text #10

Max Range: 1.3802. Min Range: -1.3802

Max Act: 0.9604. Min Act: -0.0000

Data Index: 907174 (C4 (Web Text))

Max Activating Token Index: 173

Text #11

Max Range: 1.3802. Min Range: -1.3802

Max Act: 0.9445. Min Act: -0.0000

Data Index: 47662 (C4 (Web Text))

Max Activating Token Index: 168

Text #12

Max Range: 1.3802. Min Range: -1.3802

Max Act: 0.9127. Min Act: -0.0000

Data Index: 58041 (C4 (Web Text))

Max Activating Token Index: 540

Text #13

Max Range: 1.3802. Min Range: -1.3802

Max Act: 0.9378. Min Act: -0.0000

Data Index: 803373 (C4 (Web Text))

Max Activating Token Index: 33

Text #14

Max Range: 1.3802. Min Range: -1.3802

Max Act: 0.9318. Min Act: -0.0000

Data Index: 193292 (C4 (Web Text))

Max Activating Token Index: 766

Text #15

Max Range: 1.3802. Min Range: -1.3802

Max Act: 0.9343. Min Act: -0.0000

Data Index: 719498 (C4 (Web Text))

Max Activating Token Index: 19

Text #16

Max Range: 1.3802. Min Range: -1.3802

Max Act: 0.8827. Min Act: -0.0000

Data Index: 558704 (C4 (Web Text))

Max Activating Token Index: 63

Text #17

Max Range: 1.3802. Min Range: -1.3802

Max Act: 0.8915. Min Act: -0.0000

Data Index: 1323253 (C4 (Web Text))

Max Activating Token Index: 173

Text #18

Max Range: 1.3802. Min Range: -1.3802

Max Act: 0.8724. Min Act: -0.0000

Data Index: 445991 (C4 (Web Text))

Max Activating Token Index: 280

Text #19

Max Range: 1.3802. Min Range: -1.3802

Max Act: 0.9025. Min Act: -0.0000

Data Index: 383389 (C4 (Web Text))

Max Activating Token Index: 504