Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1660 in Layer 1

Load this data into an Interactive Neuroscope

See Documentation here

Hooked Transformer Loading: HookedTransformer.from_pretrained('solu-12l')

Click the toggle to see the full text!

Text #0

Max Range: 0.0744. Min Range: -0.0744

Max Act: 0.0744. Min Act: -0.0002

Data Index: 179913 (C4 (Web Text))

Max Activating Token Index: 1015

Text #1

Max Range: 0.0744. Min Range: -0.0744

Max Act: 0.0683. Min Act: -0.0002

Data Index: 208277 (C4 (Web Text))

Max Activating Token Index: 65

Text #2

Max Range: 0.0744. Min Range: -0.0744

Max Act: 0.0674. Min Act: -0.0002

Data Index: 56198 (C4 (Web Text))

Max Activating Token Index: 876

Text #3

Max Range: 0.0744. Min Range: -0.0744

Max Act: 0.0653. Min Act: -0.0002

Data Index: 299874 (C4 (Web Text))

Max Activating Token Index: 709

Text #4

Max Range: 0.0744. Min Range: -0.0744

Max Act: 0.0649. Min Act: -0.0002

Data Index: 1317835 (C4 (Web Text))

Max Activating Token Index: 280

Text #5

Max Range: 0.0744. Min Range: -0.0744

Max Act: 0.0623. Min Act: -0.0002

Data Index: 732047 (C4 (Web Text))

Max Activating Token Index: 399

Text #6

Max Range: 0.0744. Min Range: -0.0744

Max Act: 0.0623. Min Act: -0.0002

Data Index: 1275884 (C4 (Web Text))

Max Activating Token Index: 588

Text #7

Max Range: 0.0744. Min Range: -0.0744

Max Act: 0.0626. Min Act: -0.0002

Data Index: 46386 (C4 (Web Text))

Max Activating Token Index: 828

Text #8

Max Range: 0.0744. Min Range: -0.0744

Max Act: 0.0626. Min Act: -0.0002

Data Index: 1201756 (C4 (Web Text))

Max Activating Token Index: 402

Text #9

Max Range: 0.0744. Min Range: -0.0744

Max Act: 0.0622. Min Act: -0.0002

Data Index: 724028 (C4 (Web Text))

Max Activating Token Index: 604

Text #10

Max Range: 0.0744. Min Range: -0.0744

Max Act: 0.0616. Min Act: -0.0002

Data Index: 741734 (C4 (Web Text))

Max Activating Token Index: 905

Text #11

Max Range: 0.0744. Min Range: -0.0744

Max Act: 0.0598. Min Act: -0.0002

Data Index: 1109758 (C4 (Web Text))

Max Activating Token Index: 242

Text #12

Max Range: 0.0744. Min Range: -0.0744

Max Act: 0.0592. Min Act: -0.0002

Data Index: 1090978 (C4 (Web Text))

Max Activating Token Index: 807

Text #13

Max Range: 0.0744. Min Range: -0.0744

Max Act: 0.0595. Min Act: -0.0002

Data Index: 285221 (C4 (Web Text))

Max Activating Token Index: 852

Text #14

Max Range: 0.0744. Min Range: -0.0744

Max Act: 0.0600. Min Act: -0.0003

Data Index: 407763 (C4 (Web Text))

Max Activating Token Index: 45

Text #15

Max Range: 0.0744. Min Range: -0.0744

Max Act: 0.0610. Min Act: -0.0003

Data Index: 975301 (C4 (Web Text))

Max Activating Token Index: 665

Text #16

Max Range: 0.0744. Min Range: -0.0744

Max Act: 0.0598. Min Act: -0.0002

Data Index: 1353485 (C4 (Web Text))

Max Activating Token Index: 188

Text #17

Max Range: 0.0744. Min Range: -0.0744

Max Act: 0.0583. Min Act: -0.0002

Data Index: 714359 (C4 (Web Text))

Max Activating Token Index: 306

Text #18

Max Range: 0.0744. Min Range: -0.0744

Max Act: 0.0591. Min Act: -0.0002

Data Index: 87590 (C4 (Web Text))

Max Activating Token Index: 839

Text #19

Max Range: 0.0744. Min Range: -0.0744

Max Act: 0.0602. Min Act: -0.0002

Data Index: 1276095 (C4 (Web Text))

Max Activating Token Index: 913