Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3772 in Layer 4

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 5.0726. Min Range: -5.0726

Max Act: 5.0726. Min Act: -0.0000

Data Index: 807525 (C4 (Web Text))

Max Activating Token Index: 280

Text #1

Max Range: 5.0726. Min Range: -5.0726

Max Act: 4.9435. Min Act: -0.0000

Data Index: 1148809 (C4 (Web Text))

Max Activating Token Index: 455

Text #2

Max Range: 5.0726. Min Range: -5.0726

Max Act: 4.9295. Min Act: -0.0000

Data Index: 1280315 (C4 (Web Text))

Max Activating Token Index: 226

Text #3

Max Range: 5.0726. Min Range: -5.0726

Max Act: 4.9195. Min Act: -0.0001

Data Index: 298017 (C4 (Web Text))

Max Activating Token Index: 394

Text #4

Max Range: 5.0726. Min Range: -5.0726

Max Act: 4.9221. Min Act: -0.0001

Data Index: 386192 (C4 (Web Text))

Max Activating Token Index: 613

Text #5

Max Range: 5.0726. Min Range: -5.0726

Max Act: 4.7998. Min Act: -0.0001

Data Index: 788664 (C4 (Web Text))

Max Activating Token Index: 424

Text #6

Max Range: 5.0726. Min Range: -5.0726

Max Act: 4.7424. Min Act: -0.0001

Data Index: 1154234 (C4 (Web Text))

Max Activating Token Index: 591

Text #7

Max Range: 5.0726. Min Range: -5.0726

Max Act: 4.5829. Min Act: -0.0001

Data Index: 844025 (C4 (Web Text))

Max Activating Token Index: 275

Text #8

Max Range: 5.0726. Min Range: -5.0726

Max Act: 4.7384. Min Act: -0.0001

Data Index: 656153 (C4 (Web Text))

Max Activating Token Index: 145

Text #9

Max Range: 5.0726. Min Range: -5.0726

Max Act: 4.4364. Min Act: -0.0000

Data Index: 334606 (C4 (Web Text))

Max Activating Token Index: 309

Text #10

Max Range: 5.0726. Min Range: -5.0726

Max Act: 4.6062. Min Act: -0.0001

Data Index: 1332427 (C4 (Web Text))

Max Activating Token Index: 629

Text #11

Max Range: 5.0726. Min Range: -5.0726

Max Act: 4.4723. Min Act: -0.0001

Data Index: 912869 (C4 (Web Text))

Max Activating Token Index: 449

Text #12

Max Range: 5.0726. Min Range: -5.0726

Max Act: 4.4667. Min Act: -0.0001

Data Index: 678241 (C4 (Web Text))

Max Activating Token Index: 976

Text #13

Max Range: 5.0726. Min Range: -5.0726

Max Act: 3.9887. Min Act: -0.0001

Data Index: 728243 (C4 (Web Text))

Max Activating Token Index: 642

Text #14

Max Range: 5.0726. Min Range: -5.0726

Max Act: 4.4766. Min Act: -0.0001

Data Index: 58741 (C4 (Web Text))

Max Activating Token Index: 415

Text #15

Max Range: 5.0726. Min Range: -5.0726

Max Act: 4.3177. Min Act: -0.0001

Data Index: 585758 (C4 (Web Text))

Max Activating Token Index: 602

Text #16

Max Range: 5.0726. Min Range: -5.0726

Max Act: 4.0775. Min Act: -0.0001

Data Index: 312596 (C4 (Web Text))

Max Activating Token Index: 808

Text #17

Max Range: 5.0726. Min Range: -5.0726

Max Act: 4.1692. Min Act: -0.0001

Data Index: 43184 (C4 (Web Text))

Max Activating Token Index: 877

Text #18

Max Range: 5.0726. Min Range: -5.0726

Max Act: 4.1816. Min Act: -0.0001

Data Index: 372513 (C4 (Web Text))

Max Activating Token Index: 419

Text #19

Max Range: 5.0726. Min Range: -5.0726

Max Act: 4.1842. Min Act: -0.0001

Data Index: 323556 (C4 (Web Text))

Max Activating Token Index: 897