Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 4513 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 3.8804. Min Range: -3.8804

Max Act: 3.8804. Min Act: -0.0000

Data Index: 712089 (The Pile)

Max Activating Token Index: 435

Text #1

Max Range: 3.8804. Min Range: -3.8804

Max Act: 3.7525. Min Act: -0.0000

Data Index: 748349 (The Pile)

Max Activating Token Index: 383

Text #2

Max Range: 3.8804. Min Range: -3.8804

Max Act: 3.7275. Min Act: -0.0000

Data Index: 123880 (The Pile)

Max Activating Token Index: 922

Text #3

Max Range: 3.8804. Min Range: -3.8804

Max Act: 3.6746. Min Act: -0.0000

Data Index: 383606 (The Pile)

Max Activating Token Index: 251

Text #4

Max Range: 3.8804. Min Range: -3.8804

Max Act: 3.5950. Min Act: -0.0000

Data Index: 1846782 (The Pile)

Max Activating Token Index: 342

Text #5

Max Range: 3.8804. Min Range: -3.8804

Max Act: 3.5459. Min Act: -0.0000

Data Index: 1818849 (The Pile)

Max Activating Token Index: 383

Text #6

Max Range: 3.8804. Min Range: -3.8804

Max Act: 3.6144. Min Act: -0.0000

Data Index: 1144279 (The Pile)

Max Activating Token Index: 534

Text #7

Max Range: 3.8804. Min Range: -3.8804

Max Act: 3.3694. Min Act: -0.0000

Data Index: 1747997 (The Pile)

Max Activating Token Index: 141

Text #8

Max Range: 3.8804. Min Range: -3.8804

Max Act: 3.2400. Min Act: -0.0000

Data Index: 359737 (The Pile)

Max Activating Token Index: 548

Text #9

Max Range: 3.8804. Min Range: -3.8804

Max Act: 3.0268. Min Act: -0.0000

Data Index: 1359644 (The Pile)

Max Activating Token Index: 459

Text #10

Max Range: 3.8804. Min Range: -3.8804

Max Act: 2.9079. Min Act: -0.0000

Data Index: 789507 (The Pile)

Max Activating Token Index: 879

Text #11

Max Range: 3.8804. Min Range: -3.8804

Max Act: 3.1011. Min Act: -0.0000

Data Index: 388790 (The Pile)

Max Activating Token Index: 248

Text #12

Max Range: 3.8804. Min Range: -3.8804

Max Act: 3.0748. Min Act: -0.0000

Data Index: 265253 (The Pile)

Max Activating Token Index: 110

Text #13

Max Range: 3.8804. Min Range: -3.8804

Max Act: 3.0498. Min Act: -0.0000

Data Index: 79364 (The Pile)

Max Activating Token Index: 1020

Text #14

Max Range: 3.8804. Min Range: -3.8804

Max Act: 3.0378. Min Act: -0.0000

Data Index: 1252106 (The Pile)

Max Activating Token Index: 807

Text #15

Max Range: 3.8804. Min Range: -3.8804

Max Act: 2.8979. Min Act: -0.0000

Data Index: 58573 (The Pile)

Max Activating Token Index: 790

Text #16

Max Range: 3.8804. Min Range: -3.8804

Max Act: 3.2853. Min Act: -0.0000

Data Index: 1465056 (The Pile)

Max Activating Token Index: 863

Text #17

Max Range: 3.8804. Min Range: -3.8804

Max Act: 2.9717. Min Act: -0.0000

Data Index: 872063 (The Pile)

Max Activating Token Index: 581

Text #18

Max Range: 3.8804. Min Range: -3.8804

Max Act: 2.9503. Min Act: -0.0000

Data Index: 13402 (The Pile)

Max Activating Token Index: 402

Text #19

Max Range: 3.8804. Min Range: -3.8804

Max Act: 3.0918. Min Act: -0.0000

Data Index: 541511 (The Pile)

Max Activating Token Index: 87