Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 609 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 5.0332. Min Range: -5.0332

Max Act: 4.9751. Min Act: -0.0000

Data Index: 243236 (The Pile)

Max Activating Token Index: 127

Text #1

Max Range: 5.0332. Min Range: -5.0332

Max Act: 5.0332. Min Act: -0.0000

Data Index: 677509 (The Pile)

Max Activating Token Index: 111

Text #2

Max Range: 5.0332. Min Range: -5.0332

Max Act: 4.6327. Min Act: -0.0000

Data Index: 1766216 (The Pile)

Max Activating Token Index: 90

Text #3

Max Range: 5.0332. Min Range: -5.0332

Max Act: 3.8404. Min Act: -0.0000

Data Index: 1849608 (The Pile)

Max Activating Token Index: 981

Text #4

Max Range: 5.0332. Min Range: -5.0332

Max Act: 3.8898. Min Act: -0.0000

Data Index: 1281230 (The Pile)

Max Activating Token Index: 525

Text #5

Max Range: 5.0332. Min Range: -5.0332

Max Act: 4.1469. Min Act: -0.0000

Data Index: 469635 (The Pile)

Max Activating Token Index: 422

Text #6

Max Range: 5.0332. Min Range: -5.0332

Max Act: 3.5930. Min Act: -0.0000

Data Index: 1191335 (The Pile)

Max Activating Token Index: 592

Text #7

Max Range: 5.0332. Min Range: -5.0332

Max Act: 3.2759. Min Act: -0.0000

Data Index: 507379 (The Pile)

Max Activating Token Index: 516

Text #8

Max Range: 5.0332. Min Range: -5.0332

Max Act: 3.8837. Min Act: -0.0000

Data Index: 931071 (The Pile)

Max Activating Token Index: 40

Text #9

Max Range: 5.0332. Min Range: -5.0332

Max Act: 3.5482. Min Act: -0.0000

Data Index: 941942 (The Pile)

Max Activating Token Index: 456

Text #10

Max Range: 5.0332. Min Range: -5.0332

Max Act: 3.2310. Min Act: -0.0000

Data Index: 1678032 (The Pile)

Max Activating Token Index: 854

Text #11

Max Range: 5.0332. Min Range: -5.0332

Max Act: 3.0943. Min Act: -0.0000

Data Index: 1589950 (The Pile)

Max Activating Token Index: 67

Text #12

Max Range: 5.0332. Min Range: -5.0332

Max Act: 3.0499. Min Act: -0.0000

Data Index: 1063856 (The Pile)

Max Activating Token Index: 78

Text #13

Max Range: 5.0332. Min Range: -5.0332

Max Act: 3.1287. Min Act: -0.0000

Data Index: 1497930 (The Pile)

Max Activating Token Index: 65

Text #14

Max Range: 5.0332. Min Range: -5.0332

Max Act: 2.7197. Min Act: -0.0000

Data Index: 651570 (The Pile)

Max Activating Token Index: 116

Text #15

Max Range: 5.0332. Min Range: -5.0332

Max Act: 2.7617. Min Act: -0.0000

Data Index: 9701 (The Pile)

Max Activating Token Index: 201

Text #16

Max Range: 5.0332. Min Range: -5.0332

Max Act: 2.7490. Min Act: -0.0000

Data Index: 169608 (The Pile)

Max Activating Token Index: 66

Text #17

Max Range: 5.0332. Min Range: -5.0332

Max Act: 2.8980. Min Act: -0.0000

Data Index: 1630625 (The Pile)

Max Activating Token Index: 351

Text #18

Max Range: 5.0332. Min Range: -5.0332

Max Act: 2.9319. Min Act: -0.0000

Data Index: 767673 (The Pile)

Max Activating Token Index: 986

Text #19

Max Range: 5.0332. Min Range: -5.0332

Max Act: 3.2903. Min Act: -0.0000

Data Index: 109966 (The Pile)

Max Activating Token Index: 653