Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2285 in Layer 3

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.2398. Min Range: -0.2398

Max Act: 0.2398. Min Act: -0.0001

Data Index: 1262919 (C4 (Web Text))

Max Activating Token Index: 35

Text #1

Max Range: 0.2398. Min Range: -0.2398

Max Act: 0.2207. Min Act: -0.0001

Data Index: 942272 (C4 (Web Text))

Max Activating Token Index: 104

Text #2

Max Range: 0.2398. Min Range: -0.2398

Max Act: 0.2153. Min Act: -0.0001

Data Index: 650084 (C4 (Web Text))

Max Activating Token Index: 973

Text #3

Max Range: 0.2398. Min Range: -0.2398

Max Act: 0.2194. Min Act: -0.0001

Data Index: 713929 (C4 (Web Text))

Max Activating Token Index: 74

Text #4

Max Range: 0.2398. Min Range: -0.2398

Max Act: 0.2085. Min Act: -0.0001

Data Index: 860291 (C4 (Web Text))

Max Activating Token Index: 616

Text #5

Max Range: 0.2398. Min Range: -0.2398

Max Act: 0.2002. Min Act: -0.0001

Data Index: 361375 (C4 (Web Text))

Max Activating Token Index: 226

Text #6

Max Range: 0.2398. Min Range: -0.2398

Max Act: 0.2025. Min Act: -0.0001

Data Index: 739232 (C4 (Web Text))

Max Activating Token Index: 837

Text #7

Max Range: 0.2398. Min Range: -0.2398

Max Act: 0.2094. Min Act: -0.0001

Data Index: 3645 (C4 (Web Text))

Max Activating Token Index: 540

Text #8

Max Range: 0.2398. Min Range: -0.2398

Max Act: 0.1980. Min Act: -0.0001

Data Index: 1209080 (C4 (Web Text))

Max Activating Token Index: 283

Text #9

Max Range: 0.2398. Min Range: -0.2398

Max Act: 0.1974. Min Act: -0.0001

Data Index: 1131154 (C4 (Web Text))

Max Activating Token Index: 167

Text #10

Max Range: 0.2398. Min Range: -0.2398

Max Act: 0.1920. Min Act: -0.0001

Data Index: 1236594 (C4 (Web Text))

Max Activating Token Index: 795

Text #11

Max Range: 0.2398. Min Range: -0.2398

Max Act: 0.1925. Min Act: -0.0001

Data Index: 756088 (C4 (Web Text))

Max Activating Token Index: 425

Text #12

Max Range: 0.2398. Min Range: -0.2398

Max Act: 0.1910. Min Act: -0.0001

Data Index: 4234 (C4 (Web Text))

Max Activating Token Index: 297

Text #13

Max Range: 0.2398. Min Range: -0.2398

Max Act: 0.1848. Min Act: -0.0001

Data Index: 921623 (C4 (Web Text))

Max Activating Token Index: 26

Text #14

Max Range: 0.2398. Min Range: -0.2398

Max Act: 0.1900. Min Act: -0.0001

Data Index: 147000 (C4 (Web Text))

Max Activating Token Index: 348

Text #15

Max Range: 0.2398. Min Range: -0.2398

Max Act: 0.1899. Min Act: -0.0001

Data Index: 301121 (C4 (Web Text))

Max Activating Token Index: 182

Text #16

Max Range: 0.2398. Min Range: -0.2398

Max Act: 0.1921. Min Act: -0.0001

Data Index: 1274455 (C4 (Web Text))

Max Activating Token Index: 453

Text #17

Max Range: 0.2398. Min Range: -0.2398

Max Act: 0.1915. Min Act: -0.0001

Data Index: 685340 (C4 (Web Text))

Max Activating Token Index: 232

Text #18

Max Range: 0.2398. Min Range: -0.2398

Max Act: 0.2091. Min Act: -0.0001

Data Index: 729267 (C4 (Web Text))

Max Activating Token Index: 669

Text #19

Max Range: 0.2398. Min Range: -0.2398

Max Act: 0.1843. Min Act: -0.0001

Data Index: 1318912 (C4 (Web Text))

Max Activating Token Index: 264