Model: SoLU Model: 8 Layers, 4096 Neurons per Layer

Dataset: The Pile

Neuron 1706 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-8l-pile')

Text #0

Max Range: 5.2403. Min Range: -5.2403

Max Act: 5.2403. Min Act: -0.0000

Data Index: 1342563 (The Pile)

Max Activating Token Index: 923

Text #1

Max Range: 5.2403. Min Range: -5.2403

Max Act: 4.1762. Min Act: -0.0000

Data Index: 4549 (The Pile)

Max Activating Token Index: 344

Text #2

Max Range: 5.2403. Min Range: -5.2403

Max Act: 4.0721. Min Act: -0.0000

Data Index: 1200516 (The Pile)

Max Activating Token Index: 296

Text #3

Max Range: 5.2403. Min Range: -5.2403

Max Act: 3.8428. Min Act: -0.0000

Data Index: 787740 (The Pile)

Max Activating Token Index: 602

Text #4

Max Range: 5.2403. Min Range: -5.2403

Max Act: 3.6239. Min Act: -0.0000

Data Index: 1924427 (The Pile)

Max Activating Token Index: 293

Text #5

Max Range: 5.2403. Min Range: -5.2403

Max Act: 3.5696. Min Act: -0.0000

Data Index: 741368 (The Pile)

Max Activating Token Index: 199

Text #6

Max Range: 5.2403. Min Range: -5.2403

Max Act: 3.5358. Min Act: -0.0000

Data Index: 868605 (The Pile)

Max Activating Token Index: 916

Text #7

Max Range: 5.2403. Min Range: -5.2403

Max Act: 3.4622. Min Act: -0.0000

Data Index: 926451 (The Pile)

Max Activating Token Index: 192

Text #8

Max Range: 5.2403. Min Range: -5.2403

Max Act: 3.4531. Min Act: -0.0000

Data Index: 479751 (The Pile)

Max Activating Token Index: 29

Text #9

Max Range: 5.2403. Min Range: -5.2403

Max Act: 3.3700. Min Act: -0.0000

Data Index: 1779089 (The Pile)

Max Activating Token Index: 94

Text #10

Max Range: 5.2403. Min Range: -5.2403

Max Act: 3.1237. Min Act: -0.0000

Data Index: 156136 (The Pile)

Max Activating Token Index: 299

Text #11

Max Range: 5.2403. Min Range: -5.2403

Max Act: 3.0803. Min Act: -0.0000

Data Index: 606777 (The Pile)

Max Activating Token Index: 671

Text #12

Max Range: 5.2403. Min Range: -5.2403

Max Act: 2.9022. Min Act: 0.0000

Data Index: 14286 (The Pile)

Max Activating Token Index: 14

Text #13

Max Range: 5.2403. Min Range: -5.2403

Max Act: 2.8943. Min Act: -0.0000

Data Index: 1349687 (The Pile)

Max Activating Token Index: 406

Text #14

Max Range: 5.2403. Min Range: -5.2403

Max Act: 2.8873. Min Act: -0.0000

Data Index: 1901686 (The Pile)

Max Activating Token Index: 97

Text #15

Max Range: 5.2403. Min Range: -5.2403

Max Act: 3.0098. Min Act: -0.0000

Data Index: 494197 (The Pile)

Max Activating Token Index: 485

Text #16

Max Range: 5.2403. Min Range: -5.2403

Max Act: 2.9435. Min Act: -0.0000

Data Index: 177299 (The Pile)

Max Activating Token Index: 745

Text #17

Max Range: 5.2403. Min Range: -5.2403

Max Act: 2.8741. Min Act: -0.0000

Data Index: 1520862 (The Pile)

Max Activating Token Index: 945

Text #18

Max Range: 5.2403. Min Range: -5.2403

Max Act: 2.8479. Min Act: -0.0000

Data Index: 903128 (The Pile)

Max Activating Token Index: 194

Text #19

Max Range: 5.2403. Min Range: -5.2403

Max Act: 2.8508. Min Act: -0.0000

Data Index: 823947 (The Pile)

Max Activating Token Index: 469