Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 4375 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 0.7799. Min Range: -0.7799

Max Act: 0.7799. Min Act: -0.0000

Data Index: 1052209 (The Pile)

Max Activating Token Index: 57

Text #1

Max Range: 0.7799. Min Range: -0.7799

Max Act: 0.6802. Min Act: -0.0000

Data Index: 1810320 (The Pile)

Max Activating Token Index: 964

Text #2

Max Range: 0.7799. Min Range: -0.7799

Max Act: 0.5904. Min Act: -0.0000

Data Index: 1644792 (The Pile)

Max Activating Token Index: 900

Text #3

Max Range: 0.7799. Min Range: -0.7799

Max Act: 0.5617. Min Act: -0.0000

Data Index: 1973360 (The Pile)

Max Activating Token Index: 928

Text #4

Max Range: 0.7799. Min Range: -0.7799

Max Act: 0.5635. Min Act: -0.0000

Data Index: 479437 (The Pile)

Max Activating Token Index: 169

Text #5

Max Range: 0.7799. Min Range: -0.7799

Max Act: 0.5670. Min Act: -0.0000

Data Index: 984878 (The Pile)

Max Activating Token Index: 305

Text #6

Max Range: 0.7799. Min Range: -0.7799

Max Act: 0.5441. Min Act: -0.0000

Data Index: 1590234 (The Pile)

Max Activating Token Index: 479

Text #7

Max Range: 0.7799. Min Range: -0.7799

Max Act: 0.4898. Min Act: -0.0000

Data Index: 1307411 (The Pile)

Max Activating Token Index: 423

Text #8

Max Range: 0.7799. Min Range: -0.7799

Max Act: 0.5021. Min Act: -0.0000

Data Index: 331497 (The Pile)

Max Activating Token Index: 544

Text #9

Max Range: 0.7799. Min Range: -0.7799

Max Act: 0.5155. Min Act: -0.0000

Data Index: 1804456 (The Pile)

Max Activating Token Index: 45

Text #10

Max Range: 0.7799. Min Range: -0.7799

Max Act: 0.5209. Min Act: -0.0000

Data Index: 642172 (The Pile)

Max Activating Token Index: 1019

Text #11

Max Range: 0.7799. Min Range: -0.7799

Max Act: 0.5245. Min Act: -0.0000

Data Index: 1155706 (The Pile)

Max Activating Token Index: 461

Text #12

Max Range: 0.7799. Min Range: -0.7799

Max Act: 0.5043. Min Act: -0.0000

Data Index: 1679259 (The Pile)

Max Activating Token Index: 707

Text #13

Max Range: 0.7799. Min Range: -0.7799

Max Act: 0.4990. Min Act: -0.0000

Data Index: 1966380 (The Pile)

Max Activating Token Index: 860

Text #14

Max Range: 0.7799. Min Range: -0.7799

Max Act: 0.5127. Min Act: -0.0000

Data Index: 1081042 (The Pile)

Max Activating Token Index: 467

Text #15

Max Range: 0.7799. Min Range: -0.7799

Max Act: 0.4779. Min Act: -0.0000

Data Index: 506492 (The Pile)

Max Activating Token Index: 104

Text #16

Max Range: 0.7799. Min Range: -0.7799

Max Act: 0.5036. Min Act: -0.0000

Data Index: 545707 (The Pile)

Max Activating Token Index: 30

Text #17

Max Range: 0.7799. Min Range: -0.7799

Max Act: 0.4440. Min Act: -0.0000

Data Index: 972757 (The Pile)

Max Activating Token Index: 286

Text #18

Max Range: 0.7799. Min Range: -0.7799

Max Act: 0.5217. Min Act: -0.0000

Data Index: 967572 (The Pile)

Max Activating Token Index: 755

Text #19

Max Range: 0.7799. Min Range: -0.7799

Max Act: 0.4838. Min Act: -0.0000

Data Index: 1424933 (The Pile)

Max Activating Token Index: 79