Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2304 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.2789. Min Range: -0.2789

Max Act: 0.2789. Min Act: -0.0000

Data Index: 404424 (C4 (Web Text))

Max Activating Token Index: 786

Text #1

Max Range: 0.2789. Min Range: -0.2789

Max Act: 0.1994. Min Act: -0.0000

Data Index: 447735 (C4 (Web Text))

Max Activating Token Index: 946

Text #2

Max Range: 0.2789. Min Range: -0.2789

Max Act: 0.2014. Min Act: -0.0000

Data Index: 642740 (C4 (Web Text))

Max Activating Token Index: 742

Text #3

Max Range: 0.2789. Min Range: -0.2789

Max Act: 0.1760. Min Act: -0.0000

Data Index: 237377 (C4 (Web Text))

Max Activating Token Index: 396

Text #4

Max Range: 0.2789. Min Range: -0.2789

Max Act: 0.1881. Min Act: -0.0000

Data Index: 400146 (C4 (Web Text))

Max Activating Token Index: 145

Text #5

Max Range: 0.2789. Min Range: -0.2789

Max Act: 0.1742. Min Act: -0.0000

Data Index: 460884 (C4 (Web Text))

Max Activating Token Index: 799

Text #6

Max Range: 0.2789. Min Range: -0.2789

Max Act: 0.1707. Min Act: -0.0000

Data Index: 541488 (C4 (Web Text))

Max Activating Token Index: 426

Text #7

Max Range: 0.2789. Min Range: -0.2789

Max Act: 0.1599. Min Act: -0.0000

Data Index: 495704 (C4 (Web Text))

Max Activating Token Index: 602

Text #8

Max Range: 0.2789. Min Range: -0.2789

Max Act: 0.1672. Min Act: -0.0000

Data Index: 1226547 (C4 (Web Text))

Max Activating Token Index: 860

Text #9

Max Range: 0.2789. Min Range: -0.2789

Max Act: 0.1656. Min Act: -0.0000

Data Index: 1258788 (C4 (Web Text))

Max Activating Token Index: 480

Text #10

Max Range: 0.2789. Min Range: -0.2789

Max Act: 0.1569. Min Act: -0.0000

Data Index: 174052 (C4 (Web Text))

Max Activating Token Index: 387

Text #11

Max Range: 0.2789. Min Range: -0.2789

Max Act: 0.1602. Min Act: -0.0000

Data Index: 584700 (C4 (Web Text))

Max Activating Token Index: 959

Text #12

Max Range: 0.2789. Min Range: -0.2789

Max Act: 0.1584. Min Act: -0.0000

Data Index: 773073 (C4 (Web Text))

Max Activating Token Index: 868

Text #13

Max Range: 0.2789. Min Range: -0.2789

Max Act: 0.1573. Min Act: -0.0000

Data Index: 500526 (C4 (Web Text))

Max Activating Token Index: 543

Text #14

Max Range: 0.2789. Min Range: -0.2789

Max Act: 0.1517. Min Act: -0.0000

Data Index: 877865 (C4 (Web Text))

Max Activating Token Index: 842

Text #15

Max Range: 0.2789. Min Range: -0.2789

Max Act: 0.1532. Min Act: -0.0000

Data Index: 242229 (C4 (Web Text))

Max Activating Token Index: 490

Text #16

Max Range: 0.2789. Min Range: -0.2789

Max Act: 0.1488. Min Act: -0.0000

Data Index: 775014 (C4 (Web Text))

Max Activating Token Index: 678

Text #17

Max Range: 0.2789. Min Range: -0.2789

Max Act: 0.1568. Min Act: -0.0000

Data Index: 432487 (C4 (Web Text))

Max Activating Token Index: 420

Text #18

Max Range: 0.2789. Min Range: -0.2789

Max Act: 0.1509. Min Act: -0.0000

Data Index: 1059165 (C4 (Web Text))

Max Activating Token Index: 363

Text #19

Max Range: 0.2789. Min Range: -0.2789

Max Act: 0.1412. Min Act: -0.0000

Data Index: 601641 (C4 (Web Text))

Max Activating Token Index: 1014