Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3986 in Layer 0

Load this data into an Interactive Neuroscope

See Documentation here

Hooked Transformer Loading: HookedTransformer.from_pretrained('solu-12l')

Click the toggle to see the full text!

Text #0

Max Range: 0.1055. Min Range: -0.1055

Max Act: 0.1055. Min Act: -0.0001

Data Index: 446273 (C4 (Web Text))

Max Activating Token Index: 864

Text #1

Max Range: 0.1055. Min Range: -0.1055

Max Act: 0.0932. Min Act: -0.0001

Data Index: 980379 (C4 (Web Text))

Max Activating Token Index: 164

Text #2

Max Range: 0.1055. Min Range: -0.1055

Max Act: 0.0914. Min Act: -0.0001

Data Index: 696821 (C4 (Web Text))

Max Activating Token Index: 192

Text #3

Max Range: 0.1055. Min Range: -0.1055

Max Act: 0.0896. Min Act: -0.0001

Data Index: 268914 (C4 (Web Text))

Max Activating Token Index: 778

Text #4

Max Range: 0.1055. Min Range: -0.1055

Max Act: 0.0878. Min Act: -0.0001

Data Index: 904720 (C4 (Web Text))

Max Activating Token Index: 392

Text #5

Max Range: 0.1055. Min Range: -0.1055

Max Act: 0.0890. Min Act: -0.0001

Data Index: 696262 (C4 (Web Text))

Max Activating Token Index: 826

Text #6

Max Range: 0.1055. Min Range: -0.1055

Max Act: 0.0874. Min Act: -0.0001

Data Index: 1030133 (C4 (Web Text))

Max Activating Token Index: 760

Text #7

Max Range: 0.1055. Min Range: -0.1055

Max Act: 0.0863. Min Act: -0.0001

Data Index: 228096 (C4 (Web Text))

Max Activating Token Index: 794

Text #8

Max Range: 0.1055. Min Range: -0.1055

Max Act: 0.0879. Min Act: -0.0001

Data Index: 856416 (C4 (Web Text))

Max Activating Token Index: 912

Text #9

Max Range: 0.1055. Min Range: -0.1055

Max Act: 0.0870. Min Act: -0.0001

Data Index: 215242 (C4 (Web Text))

Max Activating Token Index: 331

Text #10

Max Range: 0.1055. Min Range: -0.1055

Max Act: 0.0856. Min Act: -0.0001

Data Index: 84998 (C4 (Web Text))

Max Activating Token Index: 329

Text #11

Max Range: 0.1055. Min Range: -0.1055

Max Act: 0.0869. Min Act: -0.0001

Data Index: 238993 (C4 (Web Text))

Max Activating Token Index: 171

Text #12

Max Range: 0.1055. Min Range: -0.1055

Max Act: 0.0848. Min Act: -0.0001

Data Index: 124950 (C4 (Web Text))

Max Activating Token Index: 754

Text #13

Max Range: 0.1055. Min Range: -0.1055

Max Act: 0.0843. Min Act: -0.0001

Data Index: 358843 (C4 (Web Text))

Max Activating Token Index: 415

Text #14

Max Range: 0.1055. Min Range: -0.1055

Max Act: 0.0859. Min Act: -0.0001

Data Index: 634872 (C4 (Web Text))

Max Activating Token Index: 162

Text #15

Max Range: 0.1055. Min Range: -0.1055

Max Act: 0.0848. Min Act: -0.0001

Data Index: 974330 (C4 (Web Text))

Max Activating Token Index: 465

Text #16

Max Range: 0.1055. Min Range: -0.1055

Max Act: 0.0826. Min Act: -0.0001

Data Index: 1004694 (C4 (Web Text))

Max Activating Token Index: 560

Text #17

Max Range: 0.1055. Min Range: -0.1055

Max Act: 0.0821. Min Act: -0.0001

Data Index: 1122055 (C4 (Web Text))

Max Activating Token Index: 858

Text #18

Max Range: 0.1055. Min Range: -0.1055

Max Act: 0.0799. Min Act: -0.0001

Data Index: 1295167 (C4 (Web Text))

Max Activating Token Index: 354

Text #19

Max Range: 0.1055. Min Range: -0.1055

Max Act: 0.0781. Min Act: -0.0001

Data Index: 1033371 (C4 (Web Text))

Max Activating Token Index: 982