Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 3305 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 5.9920. Min Range: -5.9920

Max Act: 5.9920. Min Act: -0.0000

Data Index: 318928 (The Pile)

Max Activating Token Index: 705

Text #1

Max Range: 5.9920. Min Range: -5.9920

Max Act: 5.0405. Min Act: -0.0000

Data Index: 336947 (The Pile)

Max Activating Token Index: 114

Text #2

Max Range: 5.9920. Min Range: -5.9920

Max Act: 4.8335. Min Act: -0.0000

Data Index: 1698833 (The Pile)

Max Activating Token Index: 92

Text #3

Max Range: 5.9920. Min Range: -5.9920

Max Act: 4.7786. Min Act: -0.0000

Data Index: 1643103 (The Pile)

Max Activating Token Index: 284

Text #4

Max Range: 5.9920. Min Range: -5.9920

Max Act: 4.4765. Min Act: -0.0000

Data Index: 1073418 (The Pile)

Max Activating Token Index: 145

Text #5

Max Range: 5.9920. Min Range: -5.9920

Max Act: 4.3679. Min Act: -0.0000

Data Index: 1874053 (The Pile)

Max Activating Token Index: 611

Text #6

Max Range: 5.9920. Min Range: -5.9920

Max Act: 4.7020. Min Act: -0.0000

Data Index: 1208511 (The Pile)

Max Activating Token Index: 499

Text #7

Max Range: 5.9920. Min Range: -5.9920

Max Act: 4.0890. Min Act: -0.0000

Data Index: 560740 (The Pile)

Max Activating Token Index: 133

Text #8

Max Range: 5.9920. Min Range: -5.9920

Max Act: 4.1518. Min Act: -0.0000

Data Index: 1470171 (The Pile)

Max Activating Token Index: 417

Text #9

Max Range: 5.9920. Min Range: -5.9920

Max Act: 4.3394. Min Act: -0.0000

Data Index: 209987 (The Pile)

Max Activating Token Index: 150

Text #10

Max Range: 5.9920. Min Range: -5.9920

Max Act: 4.0201. Min Act: -0.0000

Data Index: 229433 (The Pile)

Max Activating Token Index: 335

Text #11

Max Range: 5.9920. Min Range: -5.9920

Max Act: 4.0710. Min Act: -0.0000

Data Index: 677928 (The Pile)

Max Activating Token Index: 169

Text #12

Max Range: 5.9920. Min Range: -5.9920

Max Act: 4.2226. Min Act: -0.0000

Data Index: 1147741 (The Pile)

Max Activating Token Index: 510

Text #13

Max Range: 5.9920. Min Range: -5.9920

Max Act: 4.2069. Min Act: -0.0000

Data Index: 129553 (The Pile)

Max Activating Token Index: 657

Text #14

Max Range: 5.9920. Min Range: -5.9920

Max Act: 4.0188. Min Act: -0.0000

Data Index: 1273408 (The Pile)

Max Activating Token Index: 285

Text #15

Max Range: 5.9920. Min Range: -5.9920

Max Act: 3.3528. Min Act: -0.0000

Data Index: 1012870 (The Pile)

Max Activating Token Index: 692

Text #16

Max Range: 5.9920. Min Range: -5.9920

Max Act: 3.6229. Min Act: -0.0000

Data Index: 1485537 (The Pile)

Max Activating Token Index: 205

Text #17

Max Range: 5.9920. Min Range: -5.9920

Max Act: 3.7283. Min Act: -0.0000

Data Index: 1041832 (The Pile)

Max Activating Token Index: 569

Text #18

Max Range: 5.9920. Min Range: -5.9920

Max Act: 4.3074. Min Act: -0.0000

Data Index: 564341 (The Pile)

Max Activating Token Index: 788

Text #19

Max Range: 5.9920. Min Range: -5.9920

Max Act: 3.7579. Min Act: -0.0000

Data Index: 1085237 (The Pile)

Max Activating Token Index: 375