Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 5698 in Layer 10

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 2.1837. Min Range: -2.1837

Max Act: 2.1793. Min Act: -0.0000

Data Index: 691133 (C4 (Web Text))

Max Activating Token Index: 67

Text #1

Max Range: 2.1837. Min Range: -2.1837

Max Act: 2.1837. Min Act: -0.0000

Data Index: 353106 (C4 (Web Text))

Max Activating Token Index: 419

Text #2

Max Range: 2.1837. Min Range: -2.1837

Max Act: 2.1590. Min Act: -0.0000

Data Index: 811185 (C4 (Web Text))

Max Activating Token Index: 342

Text #3

Max Range: 2.1837. Min Range: -2.1837

Max Act: 2.1478. Min Act: -0.0000

Data Index: 582073 (C4 (Web Text))

Max Activating Token Index: 699

Text #4

Max Range: 2.1837. Min Range: -2.1837

Max Act: 2.0678. Min Act: -0.0000

Data Index: 539229 (C4 (Web Text))

Max Activating Token Index: 402

Text #5

Max Range: 2.1837. Min Range: -2.1837

Max Act: 1.8894. Min Act: -0.0000

Data Index: 144506 (C4 (Web Text))

Max Activating Token Index: 266

Text #6

Max Range: 2.1837. Min Range: -2.1837

Max Act: 2.0295. Min Act: -0.0000

Data Index: 1070207 (C4 (Web Text))

Max Activating Token Index: 286

Text #7

Max Range: 2.1837. Min Range: -2.1837

Max Act: 1.7936. Min Act: -0.0000

Data Index: 1343604 (C4 (Web Text))

Max Activating Token Index: 669

Text #8

Max Range: 2.1837. Min Range: -2.1837

Max Act: 1.9535. Min Act: -0.0000

Data Index: 293424 (C4 (Web Text))

Max Activating Token Index: 381

Text #9

Max Range: 2.1837. Min Range: -2.1837

Max Act: 1.8502. Min Act: -0.0000

Data Index: 503112 (C4 (Web Text))

Max Activating Token Index: 503

Text #10

Max Range: 2.1837. Min Range: -2.1837

Max Act: 1.8737. Min Act: -0.0000

Data Index: 614135 (C4 (Web Text))

Max Activating Token Index: 332

Text #11

Max Range: 2.1837. Min Range: -2.1837

Max Act: 1.8511. Min Act: -0.0000

Data Index: 1268741 (C4 (Web Text))

Max Activating Token Index: 816

Text #12

Max Range: 2.1837. Min Range: -2.1837

Max Act: 1.8348. Min Act: -0.0000

Data Index: 263745 (C4 (Web Text))

Max Activating Token Index: 308

Text #13

Max Range: 2.1837. Min Range: -2.1837

Max Act: 1.8059. Min Act: -0.0000

Data Index: 1138251 (C4 (Web Text))

Max Activating Token Index: 393

Text #14

Max Range: 2.1837. Min Range: -2.1837

Max Act: 1.8843. Min Act: -0.0000

Data Index: 234664 (C4 (Web Text))

Max Activating Token Index: 510

Text #15

Max Range: 2.1837. Min Range: -2.1837

Max Act: 1.7570. Min Act: -0.0000

Data Index: 432577 (C4 (Web Text))

Max Activating Token Index: 638

Text #16

Max Range: 2.1837. Min Range: -2.1837

Max Act: 1.6883. Min Act: -0.0000

Data Index: 58664 (C4 (Web Text))

Max Activating Token Index: 517

Text #17

Max Range: 2.1837. Min Range: -2.1837

Max Act: 1.6758. Min Act: 0.0000

Data Index: 1043111 (C4 (Web Text))

Max Activating Token Index: 58

Text #18

Max Range: 2.1837. Min Range: -2.1837

Max Act: 1.7544. Min Act: -0.0000

Data Index: 265114 (C4 (Web Text))

Max Activating Token Index: 829

Text #19

Max Range: 2.1837. Min Range: -2.1837

Max Act: 1.6639. Min Act: -0.0000

Data Index: 692987 (C4 (Web Text))

Max Activating Token Index: 614