Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 1747 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 3.8820. Min Range: -3.8820

Max Act: 3.8059. Min Act: -0.0000

Data Index: 506691 (The Pile)

Max Activating Token Index: 636

Text #1

Max Range: 3.8820. Min Range: -3.8820

Max Act: 3.8069. Min Act: -0.0000

Data Index: 1481608 (The Pile)

Max Activating Token Index: 873

Text #2

Max Range: 3.8820. Min Range: -3.8820

Max Act: 3.8820. Min Act: -0.0000

Data Index: 490884 (The Pile)

Max Activating Token Index: 979

Text #3

Max Range: 3.8820. Min Range: -3.8820

Max Act: 3.5945. Min Act: -0.0000

Data Index: 755334 (The Pile)

Max Activating Token Index: 779

Text #4

Max Range: 3.8820. Min Range: -3.8820

Max Act: 3.1828. Min Act: -0.0000

Data Index: 834438 (The Pile)

Max Activating Token Index: 782

Text #5

Max Range: 3.8820. Min Range: -3.8820

Max Act: 3.3643. Min Act: -0.0000

Data Index: 289146 (The Pile)

Max Activating Token Index: 138

Text #6

Max Range: 3.8820. Min Range: -3.8820

Max Act: 3.2044. Min Act: -0.0000

Data Index: 1492084 (The Pile)

Max Activating Token Index: 73

Text #7

Max Range: 3.8820. Min Range: -3.8820

Max Act: 3.2825. Min Act: -0.0000

Data Index: 1754980 (The Pile)

Max Activating Token Index: 816

Text #8

Max Range: 3.8820. Min Range: -3.8820

Max Act: 2.9197. Min Act: -0.0000

Data Index: 1660204 (The Pile)

Max Activating Token Index: 958

Text #9

Max Range: 3.8820. Min Range: -3.8820

Max Act: 3.1747. Min Act: -0.0000

Data Index: 1466377 (The Pile)

Max Activating Token Index: 236

Text #10

Max Range: 3.8820. Min Range: -3.8820

Max Act: 2.7406. Min Act: -0.0000

Data Index: 1470276 (The Pile)

Max Activating Token Index: 1018

Text #11

Max Range: 3.8820. Min Range: -3.8820

Max Act: 2.8350. Min Act: -0.0000

Data Index: 859707 (The Pile)

Max Activating Token Index: 142

Text #12

Max Range: 3.8820. Min Range: -3.8820

Max Act: 2.8222. Min Act: -0.0000

Data Index: 131162 (The Pile)

Max Activating Token Index: 609

Text #13

Max Range: 3.8820. Min Range: -3.8820

Max Act: 2.9388. Min Act: -0.0000

Data Index: 876420 (The Pile)

Max Activating Token Index: 443

Text #14

Max Range: 3.8820. Min Range: -3.8820

Max Act: 2.5472. Min Act: -0.0000

Data Index: 1760459 (The Pile)

Max Activating Token Index: 637

Text #15

Max Range: 3.8820. Min Range: -3.8820

Max Act: 2.4208. Min Act: -0.0000

Data Index: 1935268 (The Pile)

Max Activating Token Index: 669

Text #16

Max Range: 3.8820. Min Range: -3.8820

Max Act: 2.5357. Min Act: -0.0001

Data Index: 204729 (The Pile)

Max Activating Token Index: 47

Text #17

Max Range: 3.8820. Min Range: -3.8820

Max Act: 2.3900. Min Act: -0.0000

Data Index: 193301 (The Pile)

Max Activating Token Index: 379

Text #18

Max Range: 3.8820. Min Range: -3.8820

Max Act: 2.3931. Min Act: -0.0000

Data Index: 1178447 (The Pile)

Max Activating Token Index: 865

Text #19

Max Range: 3.8820. Min Range: -3.8820

Max Act: 2.5014. Min Act: -0.0000

Data Index: 209508 (The Pile)

Max Activating Token Index: 1009