Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3703 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.2788. Min Range: -1.2788

Max Act: 1.2525. Min Act: -0.0000

Data Index: 1004110 (C4 (Web Text))

Max Activating Token Index: 188

Text #1

Max Range: 1.2788. Min Range: -1.2788

Max Act: 1.2632. Min Act: -0.0000

Data Index: 201084 (C4 (Web Text))

Max Activating Token Index: 904

Text #2

Max Range: 1.2788. Min Range: -1.2788

Max Act: 1.2788. Min Act: -0.0000

Data Index: 745972 (C4 (Web Text))

Max Activating Token Index: 737

Text #3

Max Range: 1.2788. Min Range: -1.2788

Max Act: 1.2631. Min Act: -0.0000

Data Index: 792249 (C4 (Web Text))

Max Activating Token Index: 428

Text #4

Max Range: 1.2788. Min Range: -1.2788

Max Act: 1.1748. Min Act: -0.0000

Data Index: 660417 (C4 (Web Text))

Max Activating Token Index: 886

Text #5

Max Range: 1.2788. Min Range: -1.2788

Max Act: 1.0431. Min Act: -0.0000

Data Index: 1284540 (C4 (Web Text))

Max Activating Token Index: 949

Text #6

Max Range: 1.2788. Min Range: -1.2788

Max Act: 1.1051. Min Act: -0.0000

Data Index: 406829 (C4 (Web Text))

Max Activating Token Index: 401

Text #7

Max Range: 1.2788. Min Range: -1.2788

Max Act: 1.0617. Min Act: -0.0000

Data Index: 629392 (C4 (Web Text))

Max Activating Token Index: 175

Text #8

Max Range: 1.2788. Min Range: -1.2788

Max Act: 1.0152. Min Act: -0.0000

Data Index: 487759 (C4 (Web Text))

Max Activating Token Index: 684

Text #9

Max Range: 1.2788. Min Range: -1.2788

Max Act: 1.0409. Min Act: -0.0000

Data Index: 1325733 (C4 (Web Text))

Max Activating Token Index: 742

Text #10

Max Range: 1.2788. Min Range: -1.2788

Max Act: 0.9757. Min Act: -0.0000

Data Index: 983310 (C4 (Web Text))

Max Activating Token Index: 556

Text #11

Max Range: 1.2788. Min Range: -1.2788

Max Act: 0.9716. Min Act: -0.0000

Data Index: 1442765 (Python Code)

Max Activating Token Index: 146

Text #12

Max Range: 1.2788. Min Range: -1.2788

Max Act: 0.9684. Min Act: -0.0000

Data Index: 40437 (C4 (Web Text))

Max Activating Token Index: 259

Text #13

Max Range: 1.2788. Min Range: -1.2788

Max Act: 0.9907. Min Act: -0.0000

Data Index: 1234691 (C4 (Web Text))

Max Activating Token Index: 75

Text #14

Max Range: 1.2788. Min Range: -1.2788

Max Act: 0.9689. Min Act: -0.0000

Data Index: 1403396 (Python Code)

Max Activating Token Index: 632

Text #15

Max Range: 1.2788. Min Range: -1.2788

Max Act: 0.9672. Min Act: -0.0000

Data Index: 553909 (C4 (Web Text))

Max Activating Token Index: 724

Text #16

Max Range: 1.2788. Min Range: -1.2788

Max Act: 0.9641. Min Act: -0.0000

Data Index: 417546 (C4 (Web Text))

Max Activating Token Index: 677

Text #17

Max Range: 1.2788. Min Range: -1.2788

Max Act: 0.9705. Min Act: -0.0000

Data Index: 166128 (C4 (Web Text))

Max Activating Token Index: 352

Text #18

Max Range: 1.2788. Min Range: -1.2788

Max Act: 0.9488. Min Act: -0.0000

Data Index: 837484 (C4 (Web Text))

Max Activating Token Index: 409

Text #19

Max Range: 1.2788. Min Range: -1.2788

Max Act: 0.9527. Min Act: -0.0000

Data Index: 828508 (C4 (Web Text))

Max Activating Token Index: 861