Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 5053 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 0.1527. Min Range: -0.1527

Max Act: 0.1527. Min Act: -0.0000

Data Index: 887944 (C4 (Web Text))

Max Activating Token Index: 509

Text #1

Max Range: 0.1527. Min Range: -0.1527

Max Act: 0.1272. Min Act: -0.0000

Data Index: 1171674 (C4 (Web Text))

Max Activating Token Index: 171

Text #2

Max Range: 0.1527. Min Range: -0.1527

Max Act: 0.1146. Min Act: -0.0000

Data Index: 849270 (C4 (Web Text))

Max Activating Token Index: 388

Text #3

Max Range: 0.1527. Min Range: -0.1527

Max Act: 0.1136. Min Act: -0.0000

Data Index: 1179951 (C4 (Web Text))

Max Activating Token Index: 94

Text #4

Max Range: 0.1527. Min Range: -0.1527

Max Act: 0.1084. Min Act: -0.0000

Data Index: 1575540 (Python Code)

Max Activating Token Index: 394

Text #5

Max Range: 0.1527. Min Range: -0.1527

Max Act: 0.1072. Min Act: -0.0000

Data Index: 281845 (C4 (Web Text))

Max Activating Token Index: 514

Text #6

Max Range: 0.1527. Min Range: -0.1527

Max Act: 0.0983. Min Act: -0.0000

Data Index: 537281 (C4 (Web Text))

Max Activating Token Index: 451

Text #7

Max Range: 0.1527. Min Range: -0.1527

Max Act: 0.0860. Min Act: -0.0000

Data Index: 933917 (C4 (Web Text))

Max Activating Token Index: 776

Text #8

Max Range: 0.1527. Min Range: -0.1527

Max Act: 0.0885. Min Act: -0.0000

Data Index: 1564729 (Python Code)

Max Activating Token Index: 345

Text #9

Max Range: 0.1527. Min Range: -0.1527

Max Act: 0.0880. Min Act: -0.0000

Data Index: 583532 (C4 (Web Text))

Max Activating Token Index: 892

Text #10

Max Range: 0.1527. Min Range: -0.1527

Max Act: 0.0917. Min Act: -0.0000

Data Index: 918510 (C4 (Web Text))

Max Activating Token Index: 757

Text #11

Max Range: 0.1527. Min Range: -0.1527

Max Act: 0.0721. Min Act: -0.0000

Data Index: 19332 (C4 (Web Text))

Max Activating Token Index: 802

Text #12

Max Range: 0.1527. Min Range: -0.1527

Max Act: 0.0828. Min Act: -0.0000

Data Index: 1380661 (Python Code)

Max Activating Token Index: 254

Text #13

Max Range: 0.1527. Min Range: -0.1527

Max Act: 0.0851. Min Act: -0.0000

Data Index: 25460 (C4 (Web Text))

Max Activating Token Index: 244

Text #14

Max Range: 0.1527. Min Range: -0.1527

Max Act: 0.0850. Min Act: -0.0000

Data Index: 366720 (C4 (Web Text))

Max Activating Token Index: 723

Text #15

Max Range: 0.1527. Min Range: -0.1527

Max Act: 0.0830. Min Act: -0.0000

Data Index: 1256511 (C4 (Web Text))

Max Activating Token Index: 917

Text #16

Max Range: 0.1527. Min Range: -0.1527

Max Act: 0.0795. Min Act: -0.0000

Data Index: 1365299 (Python Code)

Max Activating Token Index: 590

Text #17

Max Range: 0.1527. Min Range: -0.1527

Max Act: 0.0890. Min Act: -0.0000

Data Index: 1607793 (Python Code)

Max Activating Token Index: 874

Text #18

Max Range: 0.1527. Min Range: -0.1527

Max Act: 0.0834. Min Act: -0.0000

Data Index: 1642490 (Python Code)

Max Activating Token Index: 586

Text #19

Max Range: 0.1527. Min Range: -0.1527

Max Act: 0.0867. Min Act: -0.0000

Data Index: 180212 (C4 (Web Text))

Max Activating Token Index: 814