Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1686 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.2679. Min Range: -0.2679

Max Act: 0.2679. Min Act: -0.0000

Data Index: 1316044 (C4 (Web Text))

Max Activating Token Index: 313

Text #1

Max Range: 0.2679. Min Range: -0.2679

Max Act: 0.2610. Min Act: -0.0000

Data Index: 745169 (C4 (Web Text))

Max Activating Token Index: 629

Text #2

Max Range: 0.2679. Min Range: -0.2679

Max Act: 0.2537. Min Act: -0.0000

Data Index: 1201929 (C4 (Web Text))

Max Activating Token Index: 632

Text #3

Max Range: 0.2679. Min Range: -0.2679

Max Act: 0.2373. Min Act: -0.0000

Data Index: 1293315 (C4 (Web Text))

Max Activating Token Index: 328

Text #4

Max Range: 0.2679. Min Range: -0.2679

Max Act: 0.2275. Min Act: -0.0000

Data Index: 1049249 (C4 (Web Text))

Max Activating Token Index: 548

Text #5

Max Range: 0.2679. Min Range: -0.2679

Max Act: 0.2248. Min Act: -0.0000

Data Index: 1083276 (C4 (Web Text))

Max Activating Token Index: 707

Text #6

Max Range: 0.2679. Min Range: -0.2679

Max Act: 0.2138. Min Act: -0.0000

Data Index: 1180740 (C4 (Web Text))

Max Activating Token Index: 803

Text #7

Max Range: 0.2679. Min Range: -0.2679

Max Act: 0.2254. Min Act: -0.0000

Data Index: 754071 (C4 (Web Text))

Max Activating Token Index: 62

Text #8

Max Range: 0.2679. Min Range: -0.2679

Max Act: 0.2151. Min Act: -0.0000

Data Index: 835829 (C4 (Web Text))

Max Activating Token Index: 569

Text #9

Max Range: 0.2679. Min Range: -0.2679

Max Act: 0.2133. Min Act: -0.0000

Data Index: 388684 (C4 (Web Text))

Max Activating Token Index: 702

Text #10

Max Range: 0.2679. Min Range: -0.2679

Max Act: 0.2182. Min Act: -0.0000

Data Index: 580529 (C4 (Web Text))

Max Activating Token Index: 994

Text #11

Max Range: 0.2679. Min Range: -0.2679

Max Act: 0.2059. Min Act: -0.0000

Data Index: 67413 (C4 (Web Text))

Max Activating Token Index: 521

Text #12

Max Range: 0.2679. Min Range: -0.2679

Max Act: 0.1977. Min Act: -0.0000

Data Index: 809673 (C4 (Web Text))

Max Activating Token Index: 329

Text #13

Max Range: 0.2679. Min Range: -0.2679

Max Act: 0.2024. Min Act: -0.0000

Data Index: 916588 (C4 (Web Text))

Max Activating Token Index: 475

Text #14

Max Range: 0.2679. Min Range: -0.2679

Max Act: 0.1953. Min Act: -0.0000

Data Index: 969561 (C4 (Web Text))

Max Activating Token Index: 842

Text #15

Max Range: 0.2679. Min Range: -0.2679

Max Act: 0.2023. Min Act: -0.0000

Data Index: 794523 (C4 (Web Text))

Max Activating Token Index: 624

Text #16

Max Range: 0.2679. Min Range: -0.2679

Max Act: 0.1999. Min Act: -0.0000

Data Index: 336095 (C4 (Web Text))

Max Activating Token Index: 589

Text #17

Max Range: 0.2679. Min Range: -0.2679

Max Act: 0.2014. Min Act: -0.0000

Data Index: 559615 (C4 (Web Text))

Max Activating Token Index: 342

Text #18

Max Range: 0.2679. Min Range: -0.2679

Max Act: 0.1919. Min Act: -0.0000

Data Index: 408480 (C4 (Web Text))

Max Activating Token Index: 214

Text #19

Max Range: 0.2679. Min Range: -0.2679

Max Act: 0.1874. Min Act: -0.0000

Data Index: 345420 (C4 (Web Text))

Max Activating Token Index: 867