Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 279 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 0.9030. Min Range: -0.9030

Max Act: 0.9030. Min Act: -0.0000

Data Index: 1438512 (Python Code)

Max Activating Token Index: 889

Text #1

Max Range: 0.9030. Min Range: -0.9030

Max Act: 0.8223. Min Act: -0.0000

Data Index: 1423079 (Python Code)

Max Activating Token Index: 412

Text #2

Max Range: 0.9030. Min Range: -0.9030

Max Act: 0.8565. Min Act: -0.0000

Data Index: 1567144 (Python Code)

Max Activating Token Index: 657

Text #3

Max Range: 0.9030. Min Range: -0.9030

Max Act: 0.8473. Min Act: -0.0000

Data Index: 1407882 (Python Code)

Max Activating Token Index: 415

Text #4

Max Range: 0.9030. Min Range: -0.9030

Max Act: 0.8433. Min Act: -0.0000

Data Index: 1368066 (Python Code)

Max Activating Token Index: 533

Text #5

Max Range: 0.9030. Min Range: -0.9030

Max Act: 0.8989. Min Act: -0.0000

Data Index: 1579170 (Python Code)

Max Activating Token Index: 787

Text #6

Max Range: 0.9030. Min Range: -0.9030

Max Act: 0.7447. Min Act: -0.0000

Data Index: 1440264 (Python Code)

Max Activating Token Index: 292

Text #7

Max Range: 0.9030. Min Range: -0.9030

Max Act: 0.7804. Min Act: -0.0000

Data Index: 1492763 (Python Code)

Max Activating Token Index: 280

Text #8

Max Range: 0.9030. Min Range: -0.9030

Max Act: 0.7450. Min Act: -0.0000

Data Index: 1449117 (Python Code)

Max Activating Token Index: 1023

Text #9

Max Range: 0.9030. Min Range: -0.9030

Max Act: 0.7396. Min Act: -0.0000

Data Index: 1596224 (Python Code)

Max Activating Token Index: 760

Text #10

Max Range: 0.9030. Min Range: -0.9030

Max Act: 0.7415. Min Act: -0.0000

Data Index: 1601258 (Python Code)

Max Activating Token Index: 706

Text #11

Max Range: 0.9030. Min Range: -0.9030

Max Act: 0.6932. Min Act: -0.0000

Data Index: 1474661 (Python Code)

Max Activating Token Index: 993

Text #12

Max Range: 0.9030. Min Range: -0.9030

Max Act: 0.6774. Min Act: -0.0000

Data Index: 1654141 (Python Code)

Max Activating Token Index: 823

Text #13

Max Range: 0.9030. Min Range: -0.9030

Max Act: 0.7149. Min Act: -0.0000

Data Index: 1399385 (Python Code)

Max Activating Token Index: 861

Text #14

Max Range: 0.9030. Min Range: -0.9030

Max Act: 0.6703. Min Act: -0.0000

Data Index: 1538778 (Python Code)

Max Activating Token Index: 729

Text #15

Max Range: 0.9030. Min Range: -0.9030

Max Act: 0.6880. Min Act: -0.0000

Data Index: 1368511 (Python Code)

Max Activating Token Index: 605

Text #16

Max Range: 0.9030. Min Range: -0.9030

Max Act: 0.6439. Min Act: -0.0000

Data Index: 1484427 (Python Code)

Max Activating Token Index: 912

Text #17

Max Range: 0.9030. Min Range: -0.9030

Max Act: 0.6840. Min Act: -0.0000

Data Index: 1518752 (Python Code)

Max Activating Token Index: 441

Text #18

Max Range: 0.9030. Min Range: -0.9030

Max Act: 0.6749. Min Act: -0.0000

Data Index: 1469165 (Python Code)

Max Activating Token Index: 692

Text #19

Max Range: 0.9030. Min Range: -0.9030

Max Act: 0.6557. Min Act: -0.0000

Data Index: 1389056 (Python Code)

Max Activating Token Index: 826