Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 4580 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 3.5314. Min Range: -3.5314

Max Act: 3.2938. Min Act: -0.0000

Data Index: 631168 (The Pile)

Max Activating Token Index: 858

Text #1

Max Range: 3.5314. Min Range: -3.5314

Max Act: 3.5314. Min Act: -0.0000

Data Index: 1956997 (The Pile)

Max Activating Token Index: 669

Text #2

Max Range: 3.5314. Min Range: -3.5314

Max Act: 3.3350. Min Act: -0.0000

Data Index: 527462 (The Pile)

Max Activating Token Index: 606

Text #3

Max Range: 3.5314. Min Range: -3.5314

Max Act: 3.0715. Min Act: -0.0000

Data Index: 1680885 (The Pile)

Max Activating Token Index: 855

Text #4

Max Range: 3.5314. Min Range: -3.5314

Max Act: 3.3051. Min Act: -0.0000

Data Index: 955935 (The Pile)

Max Activating Token Index: 910

Text #5

Max Range: 3.5314. Min Range: -3.5314

Max Act: 3.0456. Min Act: -0.0000

Data Index: 523256 (The Pile)

Max Activating Token Index: 784

Text #6

Max Range: 3.5314. Min Range: -3.5314

Max Act: 3.0199. Min Act: -0.0000

Data Index: 1357273 (The Pile)

Max Activating Token Index: 378

Text #7

Max Range: 3.5314. Min Range: -3.5314

Max Act: 3.2961. Min Act: -0.0000

Data Index: 1884911 (The Pile)

Max Activating Token Index: 561

Text #8

Max Range: 3.5314. Min Range: -3.5314

Max Act: 2.8137. Min Act: -0.0000

Data Index: 708912 (The Pile)

Max Activating Token Index: 418

Text #9

Max Range: 3.5314. Min Range: -3.5314

Max Act: 2.9469. Min Act: -0.0000

Data Index: 1445070 (The Pile)

Max Activating Token Index: 941

Text #10

Max Range: 3.5314. Min Range: -3.5314

Max Act: 2.7961. Min Act: -0.0000

Data Index: 875443 (The Pile)

Max Activating Token Index: 283

Text #11

Max Range: 3.5314. Min Range: -3.5314

Max Act: 2.9086. Min Act: -0.0000

Data Index: 662038 (The Pile)

Max Activating Token Index: 489

Text #12

Max Range: 3.5314. Min Range: -3.5314

Max Act: 2.8472. Min Act: -0.0000

Data Index: 1752320 (The Pile)

Max Activating Token Index: 345

Text #13

Max Range: 3.5314. Min Range: -3.5314

Max Act: 2.8665. Min Act: -0.0000

Data Index: 1753143 (The Pile)

Max Activating Token Index: 736

Text #14

Max Range: 3.5314. Min Range: -3.5314

Max Act: 2.9877. Min Act: -0.0000

Data Index: 1736957 (The Pile)

Max Activating Token Index: 1021

Text #15

Max Range: 3.5314. Min Range: -3.5314

Max Act: 2.9509. Min Act: -0.0000

Data Index: 197737 (The Pile)

Max Activating Token Index: 865

Text #16

Max Range: 3.5314. Min Range: -3.5314

Max Act: 2.7456. Min Act: -0.0000

Data Index: 1592746 (The Pile)

Max Activating Token Index: 625

Text #17

Max Range: 3.5314. Min Range: -3.5314

Max Act: 2.7411. Min Act: -0.0000

Data Index: 1386385 (The Pile)

Max Activating Token Index: 727

Text #18

Max Range: 3.5314. Min Range: -3.5314

Max Act: 2.7005. Min Act: -0.0000

Data Index: 1913114 (The Pile)

Max Activating Token Index: 834

Text #19

Max Range: 3.5314. Min Range: -3.5314

Max Act: 2.9691. Min Act: -0.0000

Data Index: 1821610 (The Pile)

Max Activating Token Index: 687