Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 6051 in Layer 1

Load this data into an Interactive Neuroscope

See Documentation here

Hooked Transformer Loading: HookedTransformer.from_pretrained('solu-12l')

Click the toggle to see the full text!

Text #0

Max Range: 1.1091. Min Range: -1.1091

Max Act: 1.0948. Min Act: -0.0002

Data Index: 220281 (C4 (Web Text))

Max Activating Token Index: 639

Text #1

Max Range: 1.1091. Min Range: -1.1091

Max Act: 1.1091. Min Act: -0.0002

Data Index: 397482 (C4 (Web Text))

Max Activating Token Index: 553

Text #2

Max Range: 1.1091. Min Range: -1.1091

Max Act: 1.0659. Min Act: -0.0002

Data Index: 203521 (C4 (Web Text))

Max Activating Token Index: 685

Text #3

Max Range: 1.1091. Min Range: -1.1091

Max Act: 1.0331. Min Act: -0.0002

Data Index: 628162 (C4 (Web Text))

Max Activating Token Index: 781

Text #4

Max Range: 1.1091. Min Range: -1.1091

Max Act: 1.0466. Min Act: -0.0002

Data Index: 728676 (C4 (Web Text))

Max Activating Token Index: 177

Text #5

Max Range: 1.1091. Min Range: -1.1091

Max Act: 1.0090. Min Act: -0.0002

Data Index: 579032 (C4 (Web Text))

Max Activating Token Index: 50

Text #6

Max Range: 1.1091. Min Range: -1.1091

Max Act: 0.9964. Min Act: -0.0002

Data Index: 1249885 (C4 (Web Text))

Max Activating Token Index: 834

Text #7

Max Range: 1.1091. Min Range: -1.1091

Max Act: 1.0074. Min Act: -0.0002

Data Index: 962026 (C4 (Web Text))

Max Activating Token Index: 232

Text #8

Max Range: 1.1091. Min Range: -1.1091

Max Act: 0.9943. Min Act: -0.0002

Data Index: 543947 (C4 (Web Text))

Max Activating Token Index: 569

Text #9

Max Range: 1.1091. Min Range: -1.1091

Max Act: 0.9643. Min Act: -0.0002

Data Index: 81425 (C4 (Web Text))

Max Activating Token Index: 944

Text #10

Max Range: 1.1091. Min Range: -1.1091

Max Act: 0.9717. Min Act: -0.0002

Data Index: 259857 (C4 (Web Text))

Max Activating Token Index: 704

Text #11

Max Range: 1.1091. Min Range: -1.1091

Max Act: 0.9664. Min Act: -0.0002

Data Index: 247642 (C4 (Web Text))

Max Activating Token Index: 755

Text #12

Max Range: 1.1091. Min Range: -1.1091

Max Act: 0.9520. Min Act: -0.0002

Data Index: 28149 (C4 (Web Text))

Max Activating Token Index: 779

Text #13

Max Range: 1.1091. Min Range: -1.1091

Max Act: 0.9434. Min Act: -0.0002

Data Index: 848822 (C4 (Web Text))

Max Activating Token Index: 682

Text #14

Max Range: 1.1091. Min Range: -1.1091

Max Act: 0.9610. Min Act: -0.0002

Data Index: 95887 (C4 (Web Text))

Max Activating Token Index: 52

Text #15

Max Range: 1.1091. Min Range: -1.1091

Max Act: 0.9473. Min Act: -0.0002

Data Index: 1343306 (C4 (Web Text))

Max Activating Token Index: 634

Text #16

Max Range: 1.1091. Min Range: -1.1091

Max Act: 0.9500. Min Act: -0.0002

Data Index: 535053 (C4 (Web Text))

Max Activating Token Index: 584

Text #17

Max Range: 1.1091. Min Range: -1.1091

Max Act: 0.9655. Min Act: -0.0002

Data Index: 560085 (C4 (Web Text))

Max Activating Token Index: 177

Text #18

Max Range: 1.1091. Min Range: -1.1091

Max Act: 0.9480. Min Act: -0.0002

Data Index: 1235130 (C4 (Web Text))

Max Activating Token Index: 70

Text #19

Max Range: 1.1091. Min Range: -1.1091

Max Act: 0.9414. Min Act: -0.0002

Data Index: 689628 (C4 (Web Text))

Max Activating Token Index: 84