Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3033 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 3.0248. Min Range: -3.0248

Max Act: 3.0248. Min Act: -0.0000

Data Index: 977086 (C4 (Web Text))

Max Activating Token Index: 1002

Text #1

Max Range: 3.0248. Min Range: -3.0248

Max Act: 2.5365. Min Act: -0.0000

Data Index: 576261 (C4 (Web Text))

Max Activating Token Index: 392

Text #2

Max Range: 3.0248. Min Range: -3.0248

Max Act: 2.5439. Min Act: -0.0000

Data Index: 1281936 (C4 (Web Text))

Max Activating Token Index: 1021

Text #3

Max Range: 3.0248. Min Range: -3.0248

Max Act: 2.4441. Min Act: -0.0000

Data Index: 1011708 (C4 (Web Text))

Max Activating Token Index: 981

Text #4

Max Range: 3.0248. Min Range: -3.0248

Max Act: 2.3563. Min Act: -0.0000

Data Index: 699 (C4 (Web Text))

Max Activating Token Index: 128

Text #5

Max Range: 3.0248. Min Range: -3.0248

Max Act: 2.2178. Min Act: -0.0000

Data Index: 1250381 (C4 (Web Text))

Max Activating Token Index: 1019

Text #6

Max Range: 3.0248. Min Range: -3.0248

Max Act: 2.3295. Min Act: -0.0000

Data Index: 827279 (C4 (Web Text))

Max Activating Token Index: 441

Text #7

Max Range: 3.0248. Min Range: -3.0248

Max Act: 2.5168. Min Act: -0.0000

Data Index: 340963 (C4 (Web Text))

Max Activating Token Index: 855

Text #8

Max Range: 3.0248. Min Range: -3.0248

Max Act: 2.1597. Min Act: -0.0000

Data Index: 514098 (C4 (Web Text))

Max Activating Token Index: 6

Text #9

Max Range: 3.0248. Min Range: -3.0248

Max Act: 2.1995. Min Act: -0.0000

Data Index: 627565 (C4 (Web Text))

Max Activating Token Index: 275

Text #10

Max Range: 3.0248. Min Range: -3.0248

Max Act: 2.0647. Min Act: -0.0000

Data Index: 411750 (C4 (Web Text))

Max Activating Token Index: 540

Text #11

Max Range: 3.0248. Min Range: -3.0248

Max Act: 2.1984. Min Act: -0.0000

Data Index: 120002 (C4 (Web Text))

Max Activating Token Index: 205

Text #12

Max Range: 3.0248. Min Range: -3.0248

Max Act: 1.8708. Min Act: -0.0000

Data Index: 432631 (C4 (Web Text))

Max Activating Token Index: 269

Text #13

Max Range: 3.0248. Min Range: -3.0248

Max Act: 1.9786. Min Act: -0.0000

Data Index: 662314 (C4 (Web Text))

Max Activating Token Index: 638

Text #14

Max Range: 3.0248. Min Range: -3.0248

Max Act: 1.8245. Min Act: -0.0000

Data Index: 329434 (C4 (Web Text))

Max Activating Token Index: 501

Text #15

Max Range: 3.0248. Min Range: -3.0248

Max Act: 1.8697. Min Act: -0.0000

Data Index: 1239576 (C4 (Web Text))

Max Activating Token Index: 592

Text #16

Max Range: 3.0248. Min Range: -3.0248

Max Act: 1.9346. Min Act: -0.0000

Data Index: 1015998 (C4 (Web Text))

Max Activating Token Index: 787

Text #17

Max Range: 3.0248. Min Range: -3.0248

Max Act: 1.3995. Min Act: -0.0000

Data Index: 224607 (C4 (Web Text))

Max Activating Token Index: 25

Text #18

Max Range: 3.0248. Min Range: -3.0248

Max Act: 1.8154. Min Act: -0.0000

Data Index: 514301 (C4 (Web Text))

Max Activating Token Index: 727

Text #19

Max Range: 3.0248. Min Range: -3.0248

Max Act: 1.8455. Min Act: -0.0000

Data Index: 1234541 (C4 (Web Text))

Max Activating Token Index: 221