Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3711 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.0686. Min Range: -1.0686

Max Act: 1.0686. Min Act: -0.0000

Data Index: 1404885 (Python Code)

Max Activating Token Index: 620

Text #1

Max Range: 1.0686. Min Range: -1.0686

Max Act: 1.0089. Min Act: -0.0000

Data Index: 1428864 (Python Code)

Max Activating Token Index: 735

Text #2

Max Range: 1.0686. Min Range: -1.0686

Max Act: 0.9318. Min Act: -0.0000

Data Index: 1653398 (Python Code)

Max Activating Token Index: 215

Text #3

Max Range: 1.0686. Min Range: -1.0686

Max Act: 0.8469. Min Act: -0.0000

Data Index: 1628432 (Python Code)

Max Activating Token Index: 409

Text #4

Max Range: 1.0686. Min Range: -1.0686

Max Act: 0.7726. Min Act: -0.0000

Data Index: 1361188 (Python Code)

Max Activating Token Index: 439

Text #5

Max Range: 1.0686. Min Range: -1.0686

Max Act: 0.7318. Min Act: -0.0000

Data Index: 1553399 (Python Code)

Max Activating Token Index: 478

Text #6

Max Range: 1.0686. Min Range: -1.0686

Max Act: 0.7209. Min Act: -0.0000

Data Index: 1565774 (Python Code)

Max Activating Token Index: 775

Text #7

Max Range: 1.0686. Min Range: -1.0686

Max Act: 0.6981. Min Act: -0.0000

Data Index: 1517117 (Python Code)

Max Activating Token Index: 765

Text #8

Max Range: 1.0686. Min Range: -1.0686

Max Act: 0.6377. Min Act: -0.0000

Data Index: 1411805 (Python Code)

Max Activating Token Index: 321

Text #9

Max Range: 1.0686. Min Range: -1.0686

Max Act: 0.6760. Min Act: -0.0000

Data Index: 1373503 (Python Code)

Max Activating Token Index: 317

Text #10

Max Range: 1.0686. Min Range: -1.0686

Max Act: 0.6741. Min Act: -0.0000

Data Index: 1536429 (Python Code)

Max Activating Token Index: 620

Text #11

Max Range: 1.0686. Min Range: -1.0686

Max Act: 0.6707. Min Act: -0.0000

Data Index: 1446782 (Python Code)

Max Activating Token Index: 551

Text #12

Max Range: 1.0686. Min Range: -1.0686

Max Act: 0.6632. Min Act: -0.0000

Data Index: 1415671 (Python Code)

Max Activating Token Index: 460

Text #13

Max Range: 1.0686. Min Range: -1.0686

Max Act: 0.6747. Min Act: -0.0000

Data Index: 1385312 (Python Code)

Max Activating Token Index: 450

Text #14

Max Range: 1.0686. Min Range: -1.0686

Max Act: 0.6979. Min Act: -0.0000

Data Index: 1438545 (Python Code)

Max Activating Token Index: 327

Text #15

Max Range: 1.0686. Min Range: -1.0686

Max Act: 0.6201. Min Act: -0.0000

Data Index: 1449324 (Python Code)

Max Activating Token Index: 303

Text #16

Max Range: 1.0686. Min Range: -1.0686

Max Act: 0.6079. Min Act: -0.0000

Data Index: 1654871 (Python Code)

Max Activating Token Index: 302

Text #17

Max Range: 1.0686. Min Range: -1.0686

Max Act: 0.5968. Min Act: -0.0000

Data Index: 1566202 (Python Code)

Max Activating Token Index: 430

Text #18

Max Range: 1.0686. Min Range: -1.0686

Max Act: 0.5984. Min Act: -0.0000

Data Index: 1625829 (Python Code)

Max Activating Token Index: 162

Text #19

Max Range: 1.0686. Min Range: -1.0686

Max Act: 0.5726. Min Act: -0.0000

Data Index: 1481927 (Python Code)

Max Activating Token Index: 546