Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 5399 in Layer 1

Load this data into an Interactive Neuroscope

See Documentation here

Hooked Transformer Loading: HookedTransformer.from_pretrained('solu-12l')

Click the toggle to see the full text!

Text #0

Max Range: 0.9677. Min Range: -0.9677

Max Act: 0.9677. Min Act: -0.0002

Data Index: 1164834 (C4 (Web Text))

Max Activating Token Index: 408

Text #1

Max Range: 0.9677. Min Range: -0.9677

Max Act: 0.9502. Min Act: -0.0002

Data Index: 166562 (C4 (Web Text))

Max Activating Token Index: 342

Text #2

Max Range: 0.9677. Min Range: -0.9677

Max Act: 0.9095. Min Act: -0.0002

Data Index: 945705 (C4 (Web Text))

Max Activating Token Index: 336

Text #3

Max Range: 0.9677. Min Range: -0.9677

Max Act: 0.8949. Min Act: -0.0002

Data Index: 940419 (C4 (Web Text))

Max Activating Token Index: 403

Text #4

Max Range: 0.9677. Min Range: -0.9677

Max Act: 0.8772. Min Act: -0.0002

Data Index: 509155 (C4 (Web Text))

Max Activating Token Index: 662

Text #5

Max Range: 0.9677. Min Range: -0.9677

Max Act: 0.8705. Min Act: -0.0002

Data Index: 500924 (C4 (Web Text))

Max Activating Token Index: 464

Text #6

Max Range: 0.9677. Min Range: -0.9677

Max Act: 0.7900. Min Act: -0.0002

Data Index: 662395 (C4 (Web Text))

Max Activating Token Index: 189

Text #7

Max Range: 0.9677. Min Range: -0.9677

Max Act: 0.8076. Min Act: -0.0002

Data Index: 593439 (C4 (Web Text))

Max Activating Token Index: 350

Text #8

Max Range: 0.9677. Min Range: -0.9677

Max Act: 0.7750. Min Act: -0.0002

Data Index: 341983 (C4 (Web Text))

Max Activating Token Index: 547

Text #9

Max Range: 0.9677. Min Range: -0.9677

Max Act: 0.7757. Min Act: -0.0002

Data Index: 1248731 (C4 (Web Text))

Max Activating Token Index: 400

Text #10

Max Range: 0.9677. Min Range: -0.9677

Max Act: 0.7636. Min Act: -0.0002

Data Index: 378164 (C4 (Web Text))

Max Activating Token Index: 544

Text #11

Max Range: 0.9677. Min Range: -0.9677

Max Act: 0.7523. Min Act: -0.0003

Data Index: 1225043 (C4 (Web Text))

Max Activating Token Index: 234

Text #12

Max Range: 0.9677. Min Range: -0.9677

Max Act: 0.7323. Min Act: -0.0002

Data Index: 277164 (C4 (Web Text))

Max Activating Token Index: 660

Text #13

Max Range: 0.9677. Min Range: -0.9677

Max Act: 0.7247. Min Act: -0.0002

Data Index: 193099 (C4 (Web Text))

Max Activating Token Index: 397

Text #14

Max Range: 0.9677. Min Range: -0.9677

Max Act: 0.7302. Min Act: -0.0002

Data Index: 120221 (C4 (Web Text))

Max Activating Token Index: 120

Text #15

Max Range: 0.9677. Min Range: -0.9677

Max Act: 0.7375. Min Act: -0.0002

Data Index: 114403 (C4 (Web Text))

Max Activating Token Index: 267

Text #16

Max Range: 0.9677. Min Range: -0.9677

Max Act: 0.7224. Min Act: -0.0002

Data Index: 383277 (C4 (Web Text))

Max Activating Token Index: 509

Text #17

Max Range: 0.9677. Min Range: -0.9677

Max Act: 0.7453. Min Act: -0.0002

Data Index: 1176256 (C4 (Web Text))

Max Activating Token Index: 452

Text #18

Max Range: 0.9677. Min Range: -0.9677

Max Act: 0.7224. Min Act: -0.0002

Data Index: 1190716 (C4 (Web Text))

Max Activating Token Index: 909

Text #19

Max Range: 0.9677. Min Range: -0.9677

Max Act: 0.7149. Min Act: -0.0002

Data Index: 579049 (C4 (Web Text))

Max Activating Token Index: 166