Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 5291 in Layer 1

Load this data into an Interactive Neuroscope

See Documentation here

Hooked Transformer Loading: HookedTransformer.from_pretrained('solu-12l')

Click the toggle to see the full text!

Text #0

Max Range: 2.9453. Min Range: -2.9453

Max Act: 2.9453. Min Act: -0.0002

Data Index: 534101 (C4 (Web Text))

Max Activating Token Index: 44

Text #1

Max Range: 2.9453. Min Range: -2.9453

Max Act: 2.8074. Min Act: -0.0002

Data Index: 307670 (C4 (Web Text))

Max Activating Token Index: 393

Text #2

Max Range: 2.9453. Min Range: -2.9453

Max Act: 2.7069. Min Act: -0.0002

Data Index: 804329 (C4 (Web Text))

Max Activating Token Index: 813

Text #3

Max Range: 2.9453. Min Range: -2.9453

Max Act: 2.7379. Min Act: -0.0002

Data Index: 1022890 (C4 (Web Text))

Max Activating Token Index: 725

Text #4

Max Range: 2.9453. Min Range: -2.9453

Max Act: 2.6187. Min Act: -0.0002

Data Index: 1347615 (C4 (Web Text))

Max Activating Token Index: 154

Text #5

Max Range: 2.9453. Min Range: -2.9453

Max Act: 2.6492. Min Act: -0.0002

Data Index: 366827 (C4 (Web Text))

Max Activating Token Index: 822

Text #6

Max Range: 2.9453. Min Range: -2.9453

Max Act: 2.5890. Min Act: -0.0002

Data Index: 155286 (C4 (Web Text))

Max Activating Token Index: 54

Text #7

Max Range: 2.9453. Min Range: -2.9453

Max Act: 2.6670. Min Act: -0.0003

Data Index: 781206 (C4 (Web Text))

Max Activating Token Index: 106

Text #8

Max Range: 2.9453. Min Range: -2.9453

Max Act: 2.6029. Min Act: -0.0002

Data Index: 1084020 (C4 (Web Text))

Max Activating Token Index: 867

Text #9

Max Range: 2.9453. Min Range: -2.9453

Max Act: 2.5586. Min Act: -0.0002

Data Index: 151979 (C4 (Web Text))

Max Activating Token Index: 6

Text #10

Max Range: 2.9453. Min Range: -2.9453

Max Act: 2.5698. Min Act: -0.0002

Data Index: 387754 (C4 (Web Text))

Max Activating Token Index: 72

Text #11

Max Range: 2.9453. Min Range: -2.9453

Max Act: 2.4971. Min Act: -0.0003

Data Index: 172312 (C4 (Web Text))

Max Activating Token Index: 835

Text #12

Max Range: 2.9453. Min Range: -2.9453

Max Act: 2.5454. Min Act: -0.0002

Data Index: 486358 (C4 (Web Text))

Max Activating Token Index: 52

Text #13

Max Range: 2.9453. Min Range: -2.9453

Max Act: 2.5823. Min Act: -0.0002

Data Index: 64253 (C4 (Web Text))

Max Activating Token Index: 813

Text #14

Max Range: 2.9453. Min Range: -2.9453

Max Act: 2.4966. Min Act: -0.0002

Data Index: 685890 (C4 (Web Text))

Max Activating Token Index: 682

Text #15

Max Range: 2.9453. Min Range: -2.9453

Max Act: 2.4639. Min Act: -0.0002

Data Index: 796741 (C4 (Web Text))

Max Activating Token Index: 743

Text #16

Max Range: 2.9453. Min Range: -2.9453

Max Act: 2.5462. Min Act: -0.0002

Data Index: 1232269 (C4 (Web Text))

Max Activating Token Index: 862

Text #17

Max Range: 2.9453. Min Range: -2.9453

Max Act: 2.5046. Min Act: -0.0002

Data Index: 323845 (C4 (Web Text))

Max Activating Token Index: 719

Text #18

Max Range: 2.9453. Min Range: -2.9453

Max Act: 2.4616. Min Act: -0.0002

Data Index: 1001549 (C4 (Web Text))

Max Activating Token Index: 687

Text #19

Max Range: 2.9453. Min Range: -2.9453

Max Act: 2.4441. Min Act: -0.0002

Data Index: 1359585 (C4 (Web Text))

Max Activating Token Index: 61