Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4269 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.2260. Min Range: -0.2260

Max Act: 0.2021. Min Act: -0.0000

Data Index: 645713 (C4 (Web Text))

Max Activating Token Index: 864

Text #1

Max Range: 0.2260. Min Range: -0.2260

Max Act: 0.1915. Min Act: -0.0000

Data Index: 1023933 (C4 (Web Text))

Max Activating Token Index: 756

Text #2

Max Range: 0.2260. Min Range: -0.2260

Max Act: 0.2260. Min Act: -0.0000

Data Index: 931459 (C4 (Web Text))

Max Activating Token Index: 971

Text #3

Max Range: 0.2260. Min Range: -0.2260

Max Act: 0.1826. Min Act: -0.0000

Data Index: 403481 (C4 (Web Text))

Max Activating Token Index: 671

Text #4

Max Range: 0.2260. Min Range: -0.2260

Max Act: 0.1789. Min Act: -0.0000

Data Index: 1280116 (C4 (Web Text))

Max Activating Token Index: 847

Text #5

Max Range: 0.2260. Min Range: -0.2260

Max Act: 0.1767. Min Act: -0.0000

Data Index: 557011 (C4 (Web Text))

Max Activating Token Index: 106

Text #6

Max Range: 0.2260. Min Range: -0.2260

Max Act: 0.1696. Min Act: -0.0000

Data Index: 486738 (C4 (Web Text))

Max Activating Token Index: 147

Text #7

Max Range: 0.2260. Min Range: -0.2260

Max Act: 0.1749. Min Act: -0.0000

Data Index: 1279315 (C4 (Web Text))

Max Activating Token Index: 425

Text #8

Max Range: 0.2260. Min Range: -0.2260

Max Act: 0.1668. Min Act: -0.0000

Data Index: 16232 (C4 (Web Text))

Max Activating Token Index: 856

Text #9

Max Range: 0.2260. Min Range: -0.2260

Max Act: 0.1613. Min Act: -0.0000

Data Index: 551296 (C4 (Web Text))

Max Activating Token Index: 1001

Text #10

Max Range: 0.2260. Min Range: -0.2260

Max Act: 0.1637. Min Act: -0.0000

Data Index: 1183043 (C4 (Web Text))

Max Activating Token Index: 331

Text #11

Max Range: 0.2260. Min Range: -0.2260

Max Act: 0.1629. Min Act: -0.0000

Data Index: 1025458 (C4 (Web Text))

Max Activating Token Index: 780

Text #12

Max Range: 0.2260. Min Range: -0.2260

Max Act: 0.1654. Min Act: -0.0000

Data Index: 717254 (C4 (Web Text))

Max Activating Token Index: 162

Text #13

Max Range: 0.2260. Min Range: -0.2260

Max Act: 0.1646. Min Act: -0.0000

Data Index: 958218 (C4 (Web Text))

Max Activating Token Index: 336

Text #14

Max Range: 0.2260. Min Range: -0.2260

Max Act: 0.1627. Min Act: -0.0000

Data Index: 164036 (C4 (Web Text))

Max Activating Token Index: 914

Text #15

Max Range: 0.2260. Min Range: -0.2260

Max Act: 0.1644. Min Act: -0.0000

Data Index: 1077082 (C4 (Web Text))

Max Activating Token Index: 643

Text #16

Max Range: 0.2260. Min Range: -0.2260

Max Act: 0.1623. Min Act: -0.0000

Data Index: 1280259 (C4 (Web Text))

Max Activating Token Index: 276

Text #17

Max Range: 0.2260. Min Range: -0.2260

Max Act: 0.1683. Min Act: -0.0000

Data Index: 1359842 (C4 (Web Text))

Max Activating Token Index: 583

Text #18

Max Range: 0.2260. Min Range: -0.2260

Max Act: 0.1663. Min Act: -0.0000

Data Index: 454184 (C4 (Web Text))

Max Activating Token Index: 826

Text #19

Max Range: 0.2260. Min Range: -0.2260

Max Act: 0.1663. Min Act: -0.0000

Data Index: 1119408 (C4 (Web Text))

Max Activating Token Index: 618