Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1421 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.2101. Min Range: -0.2101

Max Act: 0.2101. Min Act: -0.0000

Data Index: 781075 (C4 (Web Text))

Max Activating Token Index: 717

Text #1

Max Range: 0.2101. Min Range: -0.2101

Max Act: 0.1875. Min Act: -0.0000

Data Index: 137541 (C4 (Web Text))

Max Activating Token Index: 721

Text #2

Max Range: 0.2101. Min Range: -0.2101

Max Act: 0.1757. Min Act: -0.0000

Data Index: 146826 (C4 (Web Text))

Max Activating Token Index: 933

Text #3

Max Range: 0.2101. Min Range: -0.2101

Max Act: 0.1840. Min Act: -0.0000

Data Index: 1286765 (C4 (Web Text))

Max Activating Token Index: 521

Text #4

Max Range: 0.2101. Min Range: -0.2101

Max Act: 0.1763. Min Act: -0.0000

Data Index: 1276195 (C4 (Web Text))

Max Activating Token Index: 575

Text #5

Max Range: 0.2101. Min Range: -0.2101

Max Act: 0.1753. Min Act: -0.0000

Data Index: 451443 (C4 (Web Text))

Max Activating Token Index: 540

Text #6

Max Range: 0.2101. Min Range: -0.2101

Max Act: 0.1853. Min Act: -0.0000

Data Index: 1150355 (C4 (Web Text))

Max Activating Token Index: 177

Text #7

Max Range: 0.2101. Min Range: -0.2101

Max Act: 0.1751. Min Act: -0.0000

Data Index: 198358 (C4 (Web Text))

Max Activating Token Index: 970

Text #8

Max Range: 0.2101. Min Range: -0.2101

Max Act: 0.1773. Min Act: -0.0000

Data Index: 1200446 (C4 (Web Text))

Max Activating Token Index: 553

Text #9

Max Range: 0.2101. Min Range: -0.2101

Max Act: 0.1771. Min Act: -0.0000

Data Index: 866594 (C4 (Web Text))

Max Activating Token Index: 330

Text #10

Max Range: 0.2101. Min Range: -0.2101

Max Act: 0.1672. Min Act: -0.0000

Data Index: 475702 (C4 (Web Text))

Max Activating Token Index: 1005

Text #11

Max Range: 0.2101. Min Range: -0.2101

Max Act: 0.1835. Min Act: -0.0000

Data Index: 360205 (C4 (Web Text))

Max Activating Token Index: 302

Text #12

Max Range: 0.2101. Min Range: -0.2101

Max Act: 0.1686. Min Act: -0.0000

Data Index: 1052831 (C4 (Web Text))

Max Activating Token Index: 996

Text #13

Max Range: 0.2101. Min Range: -0.2101

Max Act: 0.1600. Min Act: -0.0000

Data Index: 13770 (C4 (Web Text))

Max Activating Token Index: 562

Text #14

Max Range: 0.2101. Min Range: -0.2101

Max Act: 0.1515. Min Act: -0.0000

Data Index: 723960 (C4 (Web Text))

Max Activating Token Index: 93

Text #15

Max Range: 0.2101. Min Range: -0.2101

Max Act: 0.1501. Min Act: -0.0000

Data Index: 1170297 (C4 (Web Text))

Max Activating Token Index: 874

Text #16

Max Range: 0.2101. Min Range: -0.2101

Max Act: 0.1601. Min Act: -0.0000

Data Index: 1079590 (C4 (Web Text))

Max Activating Token Index: 272

Text #17

Max Range: 0.2101. Min Range: -0.2101

Max Act: 0.1540. Min Act: -0.0000

Data Index: 532303 (C4 (Web Text))

Max Activating Token Index: 410

Text #18

Max Range: 0.2101. Min Range: -0.2101

Max Act: 0.1536. Min Act: -0.0000

Data Index: 1076267 (C4 (Web Text))

Max Activating Token Index: 641

Text #19

Max Range: 0.2101. Min Range: -0.2101

Max Act: 0.1547. Min Act: -0.0000

Data Index: 26467 (C4 (Web Text))

Max Activating Token Index: 392