Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 857 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 0.8033. Min Range: -0.8033

Max Act: 0.8033. Min Act: -0.0000

Data Index: 1315371 (The Pile)

Max Activating Token Index: 664

Text #1

Max Range: 0.8033. Min Range: -0.8033

Max Act: 0.7225. Min Act: -0.0000

Data Index: 1484200 (The Pile)

Max Activating Token Index: 1000

Text #2

Max Range: 0.8033. Min Range: -0.8033

Max Act: 0.5730. Min Act: -0.0000

Data Index: 204445 (The Pile)

Max Activating Token Index: 848

Text #3

Max Range: 0.8033. Min Range: -0.8033

Max Act: 0.6340. Min Act: -0.0000

Data Index: 200426 (The Pile)

Max Activating Token Index: 998

Text #4

Max Range: 0.8033. Min Range: -0.8033

Max Act: 0.7963. Min Act: -0.0000

Data Index: 1126283 (The Pile)

Max Activating Token Index: 972

Text #5

Max Range: 0.8033. Min Range: -0.8033

Max Act: 0.7320. Min Act: -0.0000

Data Index: 1926722 (The Pile)

Max Activating Token Index: 923

Text #6

Max Range: 0.8033. Min Range: -0.8033

Max Act: 0.6672. Min Act: -0.0000

Data Index: 1783072 (The Pile)

Max Activating Token Index: 152

Text #7

Max Range: 0.8033. Min Range: -0.8033

Max Act: 0.7037. Min Act: -0.0000

Data Index: 1801621 (The Pile)

Max Activating Token Index: 17

Text #8

Max Range: 0.8033. Min Range: -0.8033

Max Act: 0.7504. Min Act: -0.0000

Data Index: 380751 (The Pile)

Max Activating Token Index: 678

Text #9

Max Range: 0.8033. Min Range: -0.8033

Max Act: 0.6559. Min Act: -0.0000

Data Index: 360857 (The Pile)

Max Activating Token Index: 139

Text #10

Max Range: 0.8033. Min Range: -0.8033

Max Act: 0.6155. Min Act: -0.0000

Data Index: 313453 (The Pile)

Max Activating Token Index: 863

Text #11

Max Range: 0.8033. Min Range: -0.8033

Max Act: 0.6725. Min Act: -0.0000

Data Index: 1068697 (The Pile)

Max Activating Token Index: 796

Text #12

Max Range: 0.8033. Min Range: -0.8033

Max Act: 0.6312. Min Act: -0.0000

Data Index: 1608060 (The Pile)

Max Activating Token Index: 336

Text #13

Max Range: 0.8033. Min Range: -0.8033

Max Act: 0.6622. Min Act: -0.0000

Data Index: 1856913 (The Pile)

Max Activating Token Index: 312

Text #14

Max Range: 0.8033. Min Range: -0.8033

Max Act: 0.6827. Min Act: -0.0000

Data Index: 376190 (The Pile)

Max Activating Token Index: 729

Text #15

Max Range: 0.8033. Min Range: -0.8033

Max Act: 0.6750. Min Act: -0.0000

Data Index: 1131369 (The Pile)

Max Activating Token Index: 120

Text #16

Max Range: 0.8033. Min Range: -0.8033

Max Act: 0.5782. Min Act: -0.0000

Data Index: 67102 (The Pile)

Max Activating Token Index: 511

Text #17

Max Range: 0.8033. Min Range: -0.8033

Max Act: 0.6490. Min Act: -0.0000

Data Index: 223953 (The Pile)

Max Activating Token Index: 465

Text #18

Max Range: 0.8033. Min Range: -0.8033

Max Act: 0.6441. Min Act: -0.0000

Data Index: 1949618 (The Pile)

Max Activating Token Index: 806

Text #19

Max Range: 0.8033. Min Range: -0.8033

Max Act: 0.5860. Min Act: -0.0000

Data Index: 722785 (The Pile)

Max Activating Token Index: 34