Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4301 in Layer 11

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 2.1577. Min Range: -2.1577

Max Act: 2.1067. Min Act: -0.0000

Data Index: 708591 (C4 (Web Text))

Max Activating Token Index: 748

Text #1

Max Range: 2.1577. Min Range: -2.1577

Max Act: 2.1531. Min Act: -0.0000

Data Index: 1120772 (C4 (Web Text))

Max Activating Token Index: 132

Text #2

Max Range: 2.1577. Min Range: -2.1577

Max Act: 2.1577. Min Act: -0.0000

Data Index: 425240 (C4 (Web Text))

Max Activating Token Index: 265

Text #3

Max Range: 2.1577. Min Range: -2.1577

Max Act: 2.0248. Min Act: -0.0000

Data Index: 415904 (C4 (Web Text))

Max Activating Token Index: 197

Text #4

Max Range: 2.1577. Min Range: -2.1577

Max Act: 1.8437. Min Act: -0.0000

Data Index: 987892 (C4 (Web Text))

Max Activating Token Index: 665

Text #5

Max Range: 2.1577. Min Range: -2.1577

Max Act: 1.7538. Min Act: -0.0000

Data Index: 36402 (C4 (Web Text))

Max Activating Token Index: 424

Text #6

Max Range: 2.1577. Min Range: -2.1577

Max Act: 1.7720. Min Act: -0.0000

Data Index: 1229528 (C4 (Web Text))

Max Activating Token Index: 55

Text #7

Max Range: 2.1577. Min Range: -2.1577

Max Act: 1.7595. Min Act: 0.0000

Data Index: 871570 (C4 (Web Text))

Max Activating Token Index: 784

Text #8

Max Range: 2.1577. Min Range: -2.1577

Max Act: 1.6246. Min Act: -0.0000

Data Index: 898562 (C4 (Web Text))

Max Activating Token Index: 85

Text #9

Max Range: 2.1577. Min Range: -2.1577

Max Act: 1.6221. Min Act: -0.0000

Data Index: 1141169 (C4 (Web Text))

Max Activating Token Index: 1003

Text #10

Max Range: 2.1577. Min Range: -2.1577

Max Act: 1.6569. Min Act: -0.0000

Data Index: 405508 (C4 (Web Text))

Max Activating Token Index: 487

Text #11

Max Range: 2.1577. Min Range: -2.1577

Max Act: 1.3999. Min Act: -0.0000

Data Index: 1156643 (C4 (Web Text))

Max Activating Token Index: 559

Text #12

Max Range: 2.1577. Min Range: -2.1577

Max Act: 1.4688. Min Act: -0.0000

Data Index: 25158 (C4 (Web Text))

Max Activating Token Index: 889

Text #13

Max Range: 2.1577. Min Range: -2.1577

Max Act: 1.4380. Min Act: -0.0000

Data Index: 750985 (C4 (Web Text))

Max Activating Token Index: 219

Text #14

Max Range: 2.1577. Min Range: -2.1577

Max Act: 1.4408. Min Act: -0.0000

Data Index: 487976 (C4 (Web Text))

Max Activating Token Index: 745

Text #15

Max Range: 2.1577. Min Range: -2.1577

Max Act: 1.5150. Min Act: -0.0000

Data Index: 342394 (C4 (Web Text))

Max Activating Token Index: 909

Text #16

Max Range: 2.1577. Min Range: -2.1577

Max Act: 1.3539. Min Act: -0.0000

Data Index: 485264 (C4 (Web Text))

Max Activating Token Index: 902

Text #17

Max Range: 2.1577. Min Range: -2.1577

Max Act: 1.4170. Min Act: -0.0000

Data Index: 861707 (C4 (Web Text))

Max Activating Token Index: 827

Text #18

Max Range: 2.1577. Min Range: -2.1577

Max Act: 1.4780. Min Act: -0.0000

Data Index: 975252 (C4 (Web Text))

Max Activating Token Index: 763

Text #19

Max Range: 2.1577. Min Range: -2.1577

Max Act: 1.3961. Min Act: -0.0000

Data Index: 1464284 (Python Code)

Max Activating Token Index: 4