Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 3206 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 1.2833. Min Range: -1.2833

Max Act: 1.2552. Min Act: -0.0000

Data Index: 1169407 (The Pile)

Max Activating Token Index: 5

Text #1

Max Range: 1.2833. Min Range: -1.2833

Max Act: 1.2833. Min Act: -0.0000

Data Index: 46683 (The Pile)

Max Activating Token Index: 10

Text #2

Max Range: 1.2833. Min Range: -1.2833

Max Act: 1.1566. Min Act: -0.0000

Data Index: 586363 (The Pile)

Max Activating Token Index: 219

Text #3

Max Range: 1.2833. Min Range: -1.2833

Max Act: 1.1392. Min Act: -0.0000

Data Index: 911117 (The Pile)

Max Activating Token Index: 166

Text #4

Max Range: 1.2833. Min Range: -1.2833

Max Act: 1.1250. Min Act: -0.0000

Data Index: 1631928 (The Pile)

Max Activating Token Index: 43

Text #5

Max Range: 1.2833. Min Range: -1.2833

Max Act: 1.1216. Min Act: -0.0000

Data Index: 1288036 (The Pile)

Max Activating Token Index: 4

Text #6

Max Range: 1.2833. Min Range: -1.2833

Max Act: 1.0862. Min Act: -0.0000

Data Index: 971124 (The Pile)

Max Activating Token Index: 5

Text #7

Max Range: 1.2833. Min Range: -1.2833

Max Act: 1.1460. Min Act: -0.0000

Data Index: 673948 (The Pile)

Max Activating Token Index: 923

Text #8

Max Range: 1.2833. Min Range: -1.2833

Max Act: 1.0781. Min Act: -0.0000

Data Index: 1699365 (The Pile)

Max Activating Token Index: 13

Text #9

Max Range: 1.2833. Min Range: -1.2833

Max Act: 1.1097. Min Act: -0.0000

Data Index: 1882266 (The Pile)

Max Activating Token Index: 484

Text #10

Max Range: 1.2833. Min Range: -1.2833

Max Act: 1.0671. Min Act: -0.0000

Data Index: 587697 (The Pile)

Max Activating Token Index: 3

Text #11

Max Range: 1.2833. Min Range: -1.2833

Max Act: 1.0616. Min Act: -0.0000

Data Index: 999947 (The Pile)

Max Activating Token Index: 947

Text #12

Max Range: 1.2833. Min Range: -1.2833

Max Act: 1.0531. Min Act: -0.0000

Data Index: 1848729 (The Pile)

Max Activating Token Index: 12

Text #13

Max Range: 1.2833. Min Range: -1.2833

Max Act: 1.0494. Min Act: -0.0000

Data Index: 518526 (The Pile)

Max Activating Token Index: 3

Text #14

Max Range: 1.2833. Min Range: -1.2833

Max Act: 1.0506. Min Act: -0.0000

Data Index: 1818331 (The Pile)

Max Activating Token Index: 7

Text #15

Max Range: 1.2833. Min Range: -1.2833

Max Act: 1.0370. Min Act: -0.0000

Data Index: 867749 (The Pile)

Max Activating Token Index: 8

Text #16

Max Range: 1.2833. Min Range: -1.2833

Max Act: 1.0466. Min Act: -0.0000

Data Index: 197643 (The Pile)

Max Activating Token Index: 131

Text #17

Max Range: 1.2833. Min Range: -1.2833

Max Act: 1.0217. Min Act: -0.0000

Data Index: 94844 (The Pile)

Max Activating Token Index: 85

Text #18

Max Range: 1.2833. Min Range: -1.2833

Max Act: 0.9617. Min Act: -0.0000

Data Index: 349106 (The Pile)

Max Activating Token Index: 26

Text #19

Max Range: 1.2833. Min Range: -1.2833

Max Act: 0.9934. Min Act: -0.0000

Data Index: 1355170 (The Pile)

Max Activating Token Index: 16