Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4362 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 2.4654. Min Range: -2.4654

Max Act: 2.3284. Min Act: -0.0000

Data Index: 315132 (C4 (Web Text))

Max Activating Token Index: 613

Text #1

Max Range: 2.4654. Min Range: -2.4654

Max Act: 2.1396. Min Act: -0.0000

Data Index: 1348298 (C4 (Web Text))

Max Activating Token Index: 782

Text #2

Max Range: 2.4654. Min Range: -2.4654

Max Act: 2.4654. Min Act: -0.0000

Data Index: 361154 (C4 (Web Text))

Max Activating Token Index: 617

Text #3

Max Range: 2.4654. Min Range: -2.4654

Max Act: 2.1890. Min Act: -0.0000

Data Index: 335081 (C4 (Web Text))

Max Activating Token Index: 517

Text #4

Max Range: 2.4654. Min Range: -2.4654

Max Act: 2.3389. Min Act: -0.0000

Data Index: 696623 (C4 (Web Text))

Max Activating Token Index: 829

Text #5

Max Range: 2.4654. Min Range: -2.4654

Max Act: 2.2056. Min Act: -0.0000

Data Index: 1165049 (C4 (Web Text))

Max Activating Token Index: 670

Text #6

Max Range: 2.4654. Min Range: -2.4654

Max Act: 2.1106. Min Act: -0.0000

Data Index: 861800 (C4 (Web Text))

Max Activating Token Index: 502

Text #7

Max Range: 2.4654. Min Range: -2.4654

Max Act: 2.0966. Min Act: -0.0000

Data Index: 83270 (C4 (Web Text))

Max Activating Token Index: 71

Text #8

Max Range: 2.4654. Min Range: -2.4654

Max Act: 2.0638. Min Act: -0.0000

Data Index: 762507 (C4 (Web Text))

Max Activating Token Index: 424

Text #9

Max Range: 2.4654. Min Range: -2.4654

Max Act: 2.1566. Min Act: -0.0000

Data Index: 375517 (C4 (Web Text))

Max Activating Token Index: 668

Text #10

Max Range: 2.4654. Min Range: -2.4654

Max Act: 2.0655. Min Act: -0.0000

Data Index: 1114886 (C4 (Web Text))

Max Activating Token Index: 564

Text #11

Max Range: 2.4654. Min Range: -2.4654

Max Act: 2.3222. Min Act: -0.0000

Data Index: 307152 (C4 (Web Text))

Max Activating Token Index: 763

Text #12

Max Range: 2.4654. Min Range: -2.4654

Max Act: 2.0776. Min Act: -0.0000

Data Index: 907973 (C4 (Web Text))

Max Activating Token Index: 680

Text #13

Max Range: 2.4654. Min Range: -2.4654

Max Act: 1.8988. Min Act: -0.0000

Data Index: 620146 (C4 (Web Text))

Max Activating Token Index: 851

Text #14

Max Range: 2.4654. Min Range: -2.4654

Max Act: 1.8630. Min Act: -0.0000

Data Index: 883155 (C4 (Web Text))

Max Activating Token Index: 961

Text #15

Max Range: 2.4654. Min Range: -2.4654

Max Act: 1.7993. Min Act: -0.0000

Data Index: 147902 (C4 (Web Text))

Max Activating Token Index: 206

Text #16

Max Range: 2.4654. Min Range: -2.4654

Max Act: 2.1103. Min Act: -0.0000

Data Index: 1326371 (C4 (Web Text))

Max Activating Token Index: 842

Text #17

Max Range: 2.4654. Min Range: -2.4654

Max Act: 2.0748. Min Act: -0.0000

Data Index: 1235073 (C4 (Web Text))

Max Activating Token Index: 282

Text #18

Max Range: 2.4654. Min Range: -2.4654

Max Act: 2.0690. Min Act: -0.0000

Data Index: 1072328 (C4 (Web Text))

Max Activating Token Index: 432

Text #19

Max Range: 2.4654. Min Range: -2.4654

Max Act: 1.9690. Min Act: -0.0000

Data Index: 80962 (C4 (Web Text))

Max Activating Token Index: 290