Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 5387 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.4721. Min Range: -1.4721

Max Act: 1.4721. Min Act: -0.0000

Data Index: 1564667 (Python Code)

Max Activating Token Index: 890

Text #1

Max Range: 1.4721. Min Range: -1.4721

Max Act: 1.3701. Min Act: -0.0000

Data Index: 1178680 (C4 (Web Text))

Max Activating Token Index: 717

Text #2

Max Range: 1.4721. Min Range: -1.4721

Max Act: 1.3739. Min Act: -0.0000

Data Index: 967802 (C4 (Web Text))

Max Activating Token Index: 609

Text #3

Max Range: 1.4721. Min Range: -1.4721

Max Act: 1.2491. Min Act: -0.0000

Data Index: 1513665 (Python Code)

Max Activating Token Index: 737

Text #4

Max Range: 1.4721. Min Range: -1.4721

Max Act: 1.2202. Min Act: -0.0000

Data Index: 1415849 (Python Code)

Max Activating Token Index: 1005

Text #5

Max Range: 1.4721. Min Range: -1.4721

Max Act: 1.2775. Min Act: -0.0000

Data Index: 540272 (C4 (Web Text))

Max Activating Token Index: 665

Text #6

Max Range: 1.4721. Min Range: -1.4721

Max Act: 1.2246. Min Act: -0.0000

Data Index: 1137194 (C4 (Web Text))

Max Activating Token Index: 675

Text #7

Max Range: 1.4721. Min Range: -1.4721

Max Act: 1.1748. Min Act: -0.0000

Data Index: 1254880 (C4 (Web Text))

Max Activating Token Index: 979

Text #8

Max Range: 1.4721. Min Range: -1.4721

Max Act: 1.1430. Min Act: -0.0000

Data Index: 1459376 (Python Code)

Max Activating Token Index: 810

Text #9

Max Range: 1.4721. Min Range: -1.4721

Max Act: 1.2215. Min Act: -0.0000

Data Index: 1634227 (Python Code)

Max Activating Token Index: 930

Text #10

Max Range: 1.4721. Min Range: -1.4721

Max Act: 1.0823. Min Act: -0.0000

Data Index: 1438420 (Python Code)

Max Activating Token Index: 758

Text #11

Max Range: 1.4721. Min Range: -1.4721

Max Act: 1.0508. Min Act: -0.0000

Data Index: 1435544 (Python Code)

Max Activating Token Index: 665

Text #12

Max Range: 1.4721. Min Range: -1.4721

Max Act: 1.1183. Min Act: -0.0000

Data Index: 1517333 (Python Code)

Max Activating Token Index: 761

Text #13

Max Range: 1.4721. Min Range: -1.4721

Max Act: 1.0506. Min Act: -0.0000

Data Index: 1614634 (Python Code)

Max Activating Token Index: 150

Text #14

Max Range: 1.4721. Min Range: -1.4721

Max Act: 1.0391. Min Act: -0.0000

Data Index: 881651 (C4 (Web Text))

Max Activating Token Index: 135

Text #15

Max Range: 1.4721. Min Range: -1.4721

Max Act: 1.0192. Min Act: -0.0000

Data Index: 387805 (C4 (Web Text))

Max Activating Token Index: 833

Text #16

Max Range: 1.4721. Min Range: -1.4721

Max Act: 0.9211. Min Act: -0.0000

Data Index: 313218 (C4 (Web Text))

Max Activating Token Index: 147

Text #17

Max Range: 1.4721. Min Range: -1.4721

Max Act: 0.9641. Min Act: -0.0000

Data Index: 1618165 (Python Code)

Max Activating Token Index: 833

Text #18

Max Range: 1.4721. Min Range: -1.4721

Max Act: 1.0341. Min Act: -0.0000

Data Index: 1388573 (Python Code)

Max Activating Token Index: 348

Text #19

Max Range: 1.4721. Min Range: -1.4721

Max Act: 1.0623. Min Act: -0.0000

Data Index: 1564346 (Python Code)

Max Activating Token Index: 984