Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3978 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 3.0902. Min Range: -3.0902

Max Act: 3.0902. Min Act: -0.0000

Data Index: 1116019 (C4 (Web Text))

Max Activating Token Index: 267

Text #1

Max Range: 3.0902. Min Range: -3.0902

Max Act: 2.8911. Min Act: -0.0000

Data Index: 133564 (C4 (Web Text))

Max Activating Token Index: 451

Text #2

Max Range: 3.0902. Min Range: -3.0902

Max Act: 2.6305. Min Act: -0.0000

Data Index: 939684 (C4 (Web Text))

Max Activating Token Index: 940

Text #3

Max Range: 3.0902. Min Range: -3.0902

Max Act: 2.5922. Min Act: -0.0000

Data Index: 1318647 (C4 (Web Text))

Max Activating Token Index: 244

Text #4

Max Range: 3.0902. Min Range: -3.0902

Max Act: 2.3075. Min Act: -0.0000

Data Index: 303772 (C4 (Web Text))

Max Activating Token Index: 308

Text #5

Max Range: 3.0902. Min Range: -3.0902

Max Act: 2.4250. Min Act: -0.0000

Data Index: 1153305 (C4 (Web Text))

Max Activating Token Index: 781

Text #6

Max Range: 3.0902. Min Range: -3.0902

Max Act: 2.4322. Min Act: -0.0000

Data Index: 839990 (C4 (Web Text))

Max Activating Token Index: 368

Text #7

Max Range: 3.0902. Min Range: -3.0902

Max Act: 2.3196. Min Act: -0.0000

Data Index: 482407 (C4 (Web Text))

Max Activating Token Index: 541

Text #8

Max Range: 3.0902. Min Range: -3.0902

Max Act: 2.2848. Min Act: -0.0000

Data Index: 246596 (C4 (Web Text))

Max Activating Token Index: 762

Text #9

Max Range: 3.0902. Min Range: -3.0902

Max Act: 2.2704. Min Act: -0.0000

Data Index: 1291678 (C4 (Web Text))

Max Activating Token Index: 783

Text #10

Max Range: 3.0902. Min Range: -3.0902

Max Act: 2.2385. Min Act: -0.0000

Data Index: 105770 (C4 (Web Text))

Max Activating Token Index: 306

Text #11

Max Range: 3.0902. Min Range: -3.0902

Max Act: 2.1712. Min Act: -0.0000

Data Index: 1465610 (Python Code)

Max Activating Token Index: 845

Text #12

Max Range: 3.0902. Min Range: -3.0902

Max Act: 2.1431. Min Act: -0.0000

Data Index: 684262 (C4 (Web Text))

Max Activating Token Index: 208

Text #13

Max Range: 3.0902. Min Range: -3.0902

Max Act: 2.1398. Min Act: -0.0000

Data Index: 1230493 (C4 (Web Text))

Max Activating Token Index: 683

Text #14

Max Range: 3.0902. Min Range: -3.0902

Max Act: 1.9979. Min Act: -0.0000

Data Index: 614146 (C4 (Web Text))

Max Activating Token Index: 598

Text #15

Max Range: 3.0902. Min Range: -3.0902

Max Act: 2.2214. Min Act: -0.0000

Data Index: 170753 (C4 (Web Text))

Max Activating Token Index: 118

Text #16

Max Range: 3.0902. Min Range: -3.0902

Max Act: 1.9852. Min Act: -0.0000

Data Index: 1166994 (C4 (Web Text))

Max Activating Token Index: 620

Text #17

Max Range: 3.0902. Min Range: -3.0902

Max Act: 2.2116. Min Act: -0.0000

Data Index: 832939 (C4 (Web Text))

Max Activating Token Index: 910

Text #18

Max Range: 3.0902. Min Range: -3.0902

Max Act: 1.9220. Min Act: -0.0000

Data Index: 1347493 (C4 (Web Text))

Max Activating Token Index: 161

Text #19

Max Range: 3.0902. Min Range: -3.0902

Max Act: 1.9329. Min Act: -0.0000

Data Index: 1171722 (C4 (Web Text))

Max Activating Token Index: 293