Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 3920 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 3.8353. Min Range: -3.8353

Max Act: 3.8353. Min Act: -0.0000

Data Index: 1239863 (The Pile)

Max Activating Token Index: 199

Text #1

Max Range: 3.8353. Min Range: -3.8353

Max Act: 3.8064. Min Act: -0.0000

Data Index: 442493 (The Pile)

Max Activating Token Index: 779

Text #2

Max Range: 3.8353. Min Range: -3.8353

Max Act: 3.6260. Min Act: -0.0000

Data Index: 1387094 (The Pile)

Max Activating Token Index: 232

Text #3

Max Range: 3.8353. Min Range: -3.8353

Max Act: 3.5983. Min Act: -0.0000

Data Index: 601392 (The Pile)

Max Activating Token Index: 306

Text #4

Max Range: 3.8353. Min Range: -3.8353

Max Act: 3.4828. Min Act: -0.0000

Data Index: 95265 (The Pile)

Max Activating Token Index: 410

Text #5

Max Range: 3.8353. Min Range: -3.8353

Max Act: 3.4036. Min Act: -0.0000

Data Index: 995014 (The Pile)

Max Activating Token Index: 391

Text #6

Max Range: 3.8353. Min Range: -3.8353

Max Act: 3.3538. Min Act: -0.0000

Data Index: 436590 (The Pile)

Max Activating Token Index: 213

Text #7

Max Range: 3.8353. Min Range: -3.8353

Max Act: 3.0145. Min Act: -0.0000

Data Index: 399665 (The Pile)

Max Activating Token Index: 319

Text #8

Max Range: 3.8353. Min Range: -3.8353

Max Act: 3.0947. Min Act: -0.0000

Data Index: 1239013 (The Pile)

Max Activating Token Index: 505

Text #9

Max Range: 3.8353. Min Range: -3.8353

Max Act: 2.7958. Min Act: -0.0000

Data Index: 26803 (The Pile)

Max Activating Token Index: 621

Text #10

Max Range: 3.8353. Min Range: -3.8353

Max Act: 2.9087. Min Act: -0.0000

Data Index: 56464 (The Pile)

Max Activating Token Index: 420

Text #11

Max Range: 3.8353. Min Range: -3.8353

Max Act: 3.0461. Min Act: -0.0000

Data Index: 1100511 (The Pile)

Max Activating Token Index: 1023

Text #12

Max Range: 3.8353. Min Range: -3.8353

Max Act: 3.0616. Min Act: -0.0000

Data Index: 1891271 (The Pile)

Max Activating Token Index: 342

Text #13

Max Range: 3.8353. Min Range: -3.8353

Max Act: 3.1768. Min Act: -0.0000

Data Index: 1988281 (The Pile)

Max Activating Token Index: 356

Text #14

Max Range: 3.8353. Min Range: -3.8353

Max Act: 2.8420. Min Act: -0.0000

Data Index: 990609 (The Pile)

Max Activating Token Index: 283

Text #15

Max Range: 3.8353. Min Range: -3.8353

Max Act: 2.9249. Min Act: -0.0000

Data Index: 959232 (The Pile)

Max Activating Token Index: 854

Text #16

Max Range: 3.8353. Min Range: -3.8353

Max Act: 2.9178. Min Act: -0.0000

Data Index: 1104570 (The Pile)

Max Activating Token Index: 552

Text #17

Max Range: 3.8353. Min Range: -3.8353

Max Act: 2.5444. Min Act: -0.0000

Data Index: 562131 (The Pile)

Max Activating Token Index: 713

Text #18

Max Range: 3.8353. Min Range: -3.8353

Max Act: 2.7693. Min Act: -0.0000

Data Index: 1659748 (The Pile)

Max Activating Token Index: 479

Text #19

Max Range: 3.8353. Min Range: -3.8353

Max Act: 2.6351. Min Act: -0.0000

Data Index: 894354 (The Pile)

Max Activating Token Index: 471