Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 803 in Layer 11

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.1490. Min Range: -1.1490

Max Act: 1.1490. Min Act: -0.0000

Data Index: 1066195 (C4 (Web Text))

Max Activating Token Index: 214

Text #1

Max Range: 1.1490. Min Range: -1.1490

Max Act: 1.1458. Min Act: -0.0000

Data Index: 799415 (C4 (Web Text))

Max Activating Token Index: 654

Text #2

Max Range: 1.1490. Min Range: -1.1490

Max Act: 1.0837. Min Act: -0.0000

Data Index: 1190683 (C4 (Web Text))

Max Activating Token Index: 596

Text #3

Max Range: 1.1490. Min Range: -1.1490

Max Act: 1.0573. Min Act: -0.0000

Data Index: 18444 (C4 (Web Text))

Max Activating Token Index: 954

Text #4

Max Range: 1.1490. Min Range: -1.1490

Max Act: 1.0602. Min Act: -0.0000

Data Index: 183388 (C4 (Web Text))

Max Activating Token Index: 254

Text #5

Max Range: 1.1490. Min Range: -1.1490

Max Act: 1.0258. Min Act: -0.0000

Data Index: 452079 (C4 (Web Text))

Max Activating Token Index: 174

Text #6

Max Range: 1.1490. Min Range: -1.1490

Max Act: 1.0177. Min Act: -0.0000

Data Index: 1345376 (C4 (Web Text))

Max Activating Token Index: 68

Text #7

Max Range: 1.1490. Min Range: -1.1490

Max Act: 0.9605. Min Act: 0.0000

Data Index: 1109590 (C4 (Web Text))

Max Activating Token Index: 323

Text #8

Max Range: 1.1490. Min Range: -1.1490

Max Act: 0.9487. Min Act: -0.0000

Data Index: 1092302 (C4 (Web Text))

Max Activating Token Index: 107

Text #9

Max Range: 1.1490. Min Range: -1.1490

Max Act: 0.9079. Min Act: 0.0000

Data Index: 733727 (C4 (Web Text))

Max Activating Token Index: 419

Text #10

Max Range: 1.1490. Min Range: -1.1490

Max Act: 0.9122. Min Act: -0.0000

Data Index: 690701 (C4 (Web Text))

Max Activating Token Index: 204

Text #11

Max Range: 1.1490. Min Range: -1.1490

Max Act: 0.8826. Min Act: -0.0000

Data Index: 1113286 (C4 (Web Text))

Max Activating Token Index: 718

Text #12

Max Range: 1.1490. Min Range: -1.1490

Max Act: 0.8415. Min Act: -0.0000

Data Index: 60223 (C4 (Web Text))

Max Activating Token Index: 502

Text #13

Max Range: 1.1490. Min Range: -1.1490

Max Act: 0.8666. Min Act: -0.0000

Data Index: 1067572 (C4 (Web Text))

Max Activating Token Index: 861

Text #14

Max Range: 1.1490. Min Range: -1.1490

Max Act: 0.9065. Min Act: -0.0000

Data Index: 1103590 (C4 (Web Text))

Max Activating Token Index: 656

Text #15

Max Range: 1.1490. Min Range: -1.1490

Max Act: 0.8505. Min Act: -0.0000

Data Index: 1150620 (C4 (Web Text))

Max Activating Token Index: 46

Text #16

Max Range: 1.1490. Min Range: -1.1490

Max Act: 0.9042. Min Act: 0.0000

Data Index: 1003550 (C4 (Web Text))

Max Activating Token Index: 757

Text #17

Max Range: 1.1490. Min Range: -1.1490

Max Act: 0.8430. Min Act: 0.0000

Data Index: 201117 (C4 (Web Text))

Max Activating Token Index: 510

Text #18

Max Range: 1.1490. Min Range: -1.1490

Max Act: 0.7945. Min Act: -0.0000

Data Index: 342674 (C4 (Web Text))

Max Activating Token Index: 281

Text #19

Max Range: 1.1490. Min Range: -1.1490

Max Act: 0.8117. Min Act: 0.0000

Data Index: 376524 (C4 (Web Text))

Max Activating Token Index: 520