Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3107 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 4.2782. Min Range: -4.2782

Max Act: 4.1597. Min Act: -0.0000

Data Index: 551055 (C4 (Web Text))

Max Activating Token Index: 41

Text #1

Max Range: 4.2782. Min Range: -4.2782

Max Act: 4.2782. Min Act: -0.0000

Data Index: 643884 (C4 (Web Text))

Max Activating Token Index: 703

Text #2

Max Range: 4.2782. Min Range: -4.2782

Max Act: 3.8063. Min Act: -0.0000

Data Index: 263668 (C4 (Web Text))

Max Activating Token Index: 569

Text #3

Max Range: 4.2782. Min Range: -4.2782

Max Act: 3.7841. Min Act: -0.0000

Data Index: 331985 (C4 (Web Text))

Max Activating Token Index: 978

Text #4

Max Range: 4.2782. Min Range: -4.2782

Max Act: 3.9162. Min Act: -0.0000

Data Index: 463260 (C4 (Web Text))

Max Activating Token Index: 867

Text #5

Max Range: 4.2782. Min Range: -4.2782

Max Act: 3.6124. Min Act: -0.0000

Data Index: 263906 (C4 (Web Text))

Max Activating Token Index: 857

Text #6

Max Range: 4.2782. Min Range: -4.2782

Max Act: 3.6559. Min Act: -0.0000

Data Index: 889170 (C4 (Web Text))

Max Activating Token Index: 890

Text #7

Max Range: 4.2782. Min Range: -4.2782

Max Act: 3.6488. Min Act: -0.0000

Data Index: 194577 (C4 (Web Text))

Max Activating Token Index: 963

Text #8

Max Range: 4.2782. Min Range: -4.2782

Max Act: 3.2692. Min Act: -0.0000

Data Index: 1269860 (C4 (Web Text))

Max Activating Token Index: 399

Text #9

Max Range: 4.2782. Min Range: -4.2782

Max Act: 3.5589. Min Act: -0.0000

Data Index: 489499 (C4 (Web Text))

Max Activating Token Index: 122

Text #10

Max Range: 4.2782. Min Range: -4.2782

Max Act: 3.5102. Min Act: -0.0000

Data Index: 670014 (C4 (Web Text))

Max Activating Token Index: 958

Text #11

Max Range: 4.2782. Min Range: -4.2782

Max Act: 3.5046. Min Act: -0.0000

Data Index: 386096 (C4 (Web Text))

Max Activating Token Index: 616

Text #12

Max Range: 4.2782. Min Range: -4.2782

Max Act: 3.3153. Min Act: -0.0000

Data Index: 620319 (C4 (Web Text))

Max Activating Token Index: 732

Text #13

Max Range: 4.2782. Min Range: -4.2782

Max Act: 3.3201. Min Act: -0.0000

Data Index: 134409 (C4 (Web Text))

Max Activating Token Index: 468

Text #14

Max Range: 4.2782. Min Range: -4.2782

Max Act: 3.3079. Min Act: -0.0000

Data Index: 66714 (C4 (Web Text))

Max Activating Token Index: 390

Text #15

Max Range: 4.2782. Min Range: -4.2782

Max Act: 3.1096. Min Act: -0.0000

Data Index: 308848 (C4 (Web Text))

Max Activating Token Index: 782

Text #16

Max Range: 4.2782. Min Range: -4.2782

Max Act: 3.0734. Min Act: -0.0000

Data Index: 1131326 (C4 (Web Text))

Max Activating Token Index: 560

Text #17

Max Range: 4.2782. Min Range: -4.2782

Max Act: 3.2332. Min Act: -0.0000

Data Index: 139233 (C4 (Web Text))

Max Activating Token Index: 776

Text #18

Max Range: 4.2782. Min Range: -4.2782

Max Act: 3.0755. Min Act: -0.0000

Data Index: 836331 (C4 (Web Text))

Max Activating Token Index: 532

Text #19

Max Range: 4.2782. Min Range: -4.2782

Max Act: 3.0335. Min Act: -0.0000

Data Index: 449352 (C4 (Web Text))

Max Activating Token Index: 960