Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3399 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 3.2431. Min Range: -3.2431

Max Act: 3.2431. Min Act: -0.0000

Data Index: 422829 (C4 (Web Text))

Max Activating Token Index: 705

Text #1

Max Range: 3.2431. Min Range: -3.2431

Max Act: 2.3921. Min Act: -0.0000

Data Index: 916628 (C4 (Web Text))

Max Activating Token Index: 553

Text #2

Max Range: 3.2431. Min Range: -3.2431

Max Act: 2.3129. Min Act: -0.0000

Data Index: 155297 (C4 (Web Text))

Max Activating Token Index: 600

Text #3

Max Range: 3.2431. Min Range: -3.2431

Max Act: 2.2423. Min Act: -0.0000

Data Index: 640176 (C4 (Web Text))

Max Activating Token Index: 230

Text #4

Max Range: 3.2431. Min Range: -3.2431

Max Act: 2.1325. Min Act: -0.0000

Data Index: 1221981 (C4 (Web Text))

Max Activating Token Index: 752

Text #5

Max Range: 3.2431. Min Range: -3.2431

Max Act: 2.0999. Min Act: -0.0000

Data Index: 157461 (C4 (Web Text))

Max Activating Token Index: 949

Text #6

Max Range: 3.2431. Min Range: -3.2431

Max Act: 2.1993. Min Act: -0.0000

Data Index: 1170775 (C4 (Web Text))

Max Activating Token Index: 111

Text #7

Max Range: 3.2431. Min Range: -3.2431

Max Act: 2.2742. Min Act: -0.0000

Data Index: 892112 (C4 (Web Text))

Max Activating Token Index: 26

Text #8

Max Range: 3.2431. Min Range: -3.2431

Max Act: 2.1659. Min Act: -0.0000

Data Index: 833132 (C4 (Web Text))

Max Activating Token Index: 39

Text #9

Max Range: 3.2431. Min Range: -3.2431

Max Act: 2.1649. Min Act: -0.0000

Data Index: 695869 (C4 (Web Text))

Max Activating Token Index: 878

Text #10

Max Range: 3.2431. Min Range: -3.2431

Max Act: 2.1306. Min Act: -0.0000

Data Index: 709793 (C4 (Web Text))

Max Activating Token Index: 534

Text #11

Max Range: 3.2431. Min Range: -3.2431

Max Act: 2.1246. Min Act: -0.0000

Data Index: 216463 (C4 (Web Text))

Max Activating Token Index: 327

Text #12

Max Range: 3.2431. Min Range: -3.2431

Max Act: 2.2089. Min Act: -0.0000

Data Index: 1018436 (C4 (Web Text))

Max Activating Token Index: 137

Text #13

Max Range: 3.2431. Min Range: -3.2431

Max Act: 1.9888. Min Act: -0.0000

Data Index: 875312 (C4 (Web Text))

Max Activating Token Index: 30

Text #14

Max Range: 3.2431. Min Range: -3.2431

Max Act: 1.9828. Min Act: -0.0000

Data Index: 308174 (C4 (Web Text))

Max Activating Token Index: 57

Text #15

Max Range: 3.2431. Min Range: -3.2431

Max Act: 2.0825. Min Act: -0.0000

Data Index: 132630 (C4 (Web Text))

Max Activating Token Index: 57

Text #16

Max Range: 3.2431. Min Range: -3.2431

Max Act: 2.0772. Min Act: -0.0000

Data Index: 14519 (C4 (Web Text))

Max Activating Token Index: 59

Text #17

Max Range: 3.2431. Min Range: -3.2431

Max Act: 2.0617. Min Act: -0.0000

Data Index: 581328 (C4 (Web Text))

Max Activating Token Index: 802

Text #18

Max Range: 3.2431. Min Range: -3.2431

Max Act: 2.0641. Min Act: -0.0000

Data Index: 1012655 (C4 (Web Text))

Max Activating Token Index: 562

Text #19

Max Range: 3.2431. Min Range: -3.2431

Max Act: 2.0355. Min Act: -0.0000

Data Index: 665154 (C4 (Web Text))

Max Activating Token Index: 1001