Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1996 in Layer 0

Load this data into an Interactive Neuroscope

See Documentation here

Hooked Transformer Loading: HookedTransformer.from_pretrained('solu-12l')

Click the toggle to see the full text!

Text #0

Max Range: 0.5667. Min Range: -0.5667

Max Act: 0.5667. Min Act: -0.0001

Data Index: 86462 (C4 (Web Text))

Max Activating Token Index: 120

Text #1

Max Range: 0.5667. Min Range: -0.5667

Max Act: 0.5493. Min Act: -0.0001

Data Index: 371084 (C4 (Web Text))

Max Activating Token Index: 335

Text #2

Max Range: 0.5667. Min Range: -0.5667

Max Act: 0.5502. Min Act: -0.0001

Data Index: 503999 (C4 (Web Text))

Max Activating Token Index: 82

Text #3

Max Range: 0.5667. Min Range: -0.5667

Max Act: 0.5506. Min Act: -0.0001

Data Index: 201656 (C4 (Web Text))

Max Activating Token Index: 1004

Text #4

Max Range: 0.5667. Min Range: -0.5667

Max Act: 0.5469. Min Act: -0.0001

Data Index: 1156097 (C4 (Web Text))

Max Activating Token Index: 74

Text #5

Max Range: 0.5667. Min Range: -0.5667

Max Act: 0.5322. Min Act: -0.0001

Data Index: 1261488 (C4 (Web Text))

Max Activating Token Index: 930

Text #6

Max Range: 0.5667. Min Range: -0.5667

Max Act: 0.5107. Min Act: -0.0001

Data Index: 155510 (C4 (Web Text))

Max Activating Token Index: 785

Text #7

Max Range: 0.5667. Min Range: -0.5667

Max Act: 0.5189. Min Act: -0.0001

Data Index: 988992 (C4 (Web Text))

Max Activating Token Index: 840

Text #8

Max Range: 0.5667. Min Range: -0.5667

Max Act: 0.5140. Min Act: -0.0001

Data Index: 426266 (C4 (Web Text))

Max Activating Token Index: 155

Text #9

Max Range: 0.5667. Min Range: -0.5667

Max Act: 0.5185. Min Act: -0.0001

Data Index: 667659 (C4 (Web Text))

Max Activating Token Index: 776

Text #10

Max Range: 0.5667. Min Range: -0.5667

Max Act: 0.5152. Min Act: -0.0001

Data Index: 1130799 (C4 (Web Text))

Max Activating Token Index: 343

Text #11

Max Range: 0.5667. Min Range: -0.5667

Max Act: 0.5160. Min Act: -0.0001

Data Index: 511615 (C4 (Web Text))

Max Activating Token Index: 136

Text #12

Max Range: 0.5667. Min Range: -0.5667

Max Act: 0.5012. Min Act: -0.0001

Data Index: 1532267 (Python Code)

Max Activating Token Index: 1023

Text #13

Max Range: 0.5667. Min Range: -0.5667

Max Act: 0.5000. Min Act: -0.0001

Data Index: 35037 (C4 (Web Text))

Max Activating Token Index: 72

Text #14

Max Range: 0.5667. Min Range: -0.5667

Max Act: 0.4978. Min Act: -0.0001

Data Index: 493204 (C4 (Web Text))

Max Activating Token Index: 636

Text #15

Max Range: 0.5667. Min Range: -0.5667

Max Act: 0.5102. Min Act: -0.0001

Data Index: 1489794 (Python Code)

Max Activating Token Index: 65

Text #16

Max Range: 0.5667. Min Range: -0.5667

Max Act: 0.4974. Min Act: -0.0001

Data Index: 189321 (C4 (Web Text))

Max Activating Token Index: 427

Text #17

Max Range: 0.5667. Min Range: -0.5667

Max Act: 0.5034. Min Act: -0.0001

Data Index: 547824 (C4 (Web Text))

Max Activating Token Index: 390

Text #18

Max Range: 0.5667. Min Range: -0.5667

Max Act: 0.4995. Min Act: -0.0001

Data Index: 387106 (C4 (Web Text))

Max Activating Token Index: 859

Text #19

Max Range: 0.5667. Min Range: -0.5667

Max Act: 0.4896. Min Act: -0.0001

Data Index: 448287 (C4 (Web Text))

Max Activating Token Index: 792