Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 463 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.2238. Min Range: -0.2238

Max Act: 0.2238. Min Act: -0.0000

Data Index: 1199873 (C4 (Web Text))

Max Activating Token Index: 672

Text #1

Max Range: 0.2238. Min Range: -0.2238

Max Act: 0.2087. Min Act: -0.0000

Data Index: 28998 (C4 (Web Text))

Max Activating Token Index: 57

Text #2

Max Range: 0.2238. Min Range: -0.2238

Max Act: 0.2061. Min Act: -0.0000

Data Index: 652195 (C4 (Web Text))

Max Activating Token Index: 691

Text #3

Max Range: 0.2238. Min Range: -0.2238

Max Act: 0.2072. Min Act: -0.0000

Data Index: 704847 (C4 (Web Text))

Max Activating Token Index: 978

Text #4

Max Range: 0.2238. Min Range: -0.2238

Max Act: 0.2038. Min Act: -0.0000

Data Index: 497491 (C4 (Web Text))

Max Activating Token Index: 385

Text #5

Max Range: 0.2238. Min Range: -0.2238

Max Act: 0.1930. Min Act: -0.0000

Data Index: 302476 (C4 (Web Text))

Max Activating Token Index: 599

Text #6

Max Range: 0.2238. Min Range: -0.2238

Max Act: 0.1916. Min Act: -0.0000

Data Index: 516273 (C4 (Web Text))

Max Activating Token Index: 59

Text #7

Max Range: 0.2238. Min Range: -0.2238

Max Act: 0.1877. Min Act: -0.0000

Data Index: 1218979 (C4 (Web Text))

Max Activating Token Index: 935

Text #8

Max Range: 0.2238. Min Range: -0.2238

Max Act: 0.1795. Min Act: -0.0000

Data Index: 1284232 (C4 (Web Text))

Max Activating Token Index: 200

Text #9

Max Range: 0.2238. Min Range: -0.2238

Max Act: 0.1769. Min Act: -0.0000

Data Index: 397409 (C4 (Web Text))

Max Activating Token Index: 621

Text #10

Max Range: 0.2238. Min Range: -0.2238

Max Act: 0.1768. Min Act: -0.0000

Data Index: 45691 (C4 (Web Text))

Max Activating Token Index: 912

Text #11

Max Range: 0.2238. Min Range: -0.2238

Max Act: 0.1891. Min Act: -0.0000

Data Index: 1236413 (C4 (Web Text))

Max Activating Token Index: 652

Text #12

Max Range: 0.2238. Min Range: -0.2238

Max Act: 0.1790. Min Act: -0.0000

Data Index: 486761 (C4 (Web Text))

Max Activating Token Index: 1016

Text #13

Max Range: 0.2238. Min Range: -0.2238

Max Act: 0.1788. Min Act: -0.0000

Data Index: 1179971 (C4 (Web Text))

Max Activating Token Index: 473

Text #14

Max Range: 0.2238. Min Range: -0.2238

Max Act: 0.1711. Min Act: -0.0000

Data Index: 339903 (C4 (Web Text))

Max Activating Token Index: 537

Text #15

Max Range: 0.2238. Min Range: -0.2238

Max Act: 0.1816. Min Act: -0.0000

Data Index: 1108805 (C4 (Web Text))

Max Activating Token Index: 870

Text #16

Max Range: 0.2238. Min Range: -0.2238

Max Act: 0.1758. Min Act: -0.0000

Data Index: 290335 (C4 (Web Text))

Max Activating Token Index: 813

Text #17

Max Range: 0.2238. Min Range: -0.2238

Max Act: 0.1754. Min Act: -0.0000

Data Index: 1289481 (C4 (Web Text))

Max Activating Token Index: 327

Text #18

Max Range: 0.2238. Min Range: -0.2238

Max Act: 0.1739. Min Act: -0.0000

Data Index: 135943 (C4 (Web Text))

Max Activating Token Index: 675

Text #19

Max Range: 0.2238. Min Range: -0.2238

Max Act: 0.1730. Min Act: -0.0000

Data Index: 308429 (C4 (Web Text))

Max Activating Token Index: 181