Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 45 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 3.5341. Min Range: -3.5341

Max Act: 3.5341. Min Act: -0.0000

Data Index: 1608482 (The Pile)

Max Activating Token Index: 464

Text #1

Max Range: 3.5341. Min Range: -3.5341

Max Act: 3.2618. Min Act: -0.0000

Data Index: 529301 (The Pile)

Max Activating Token Index: 235

Text #2

Max Range: 3.5341. Min Range: -3.5341

Max Act: 3.2082. Min Act: -0.0000

Data Index: 337739 (The Pile)

Max Activating Token Index: 632

Text #3

Max Range: 3.5341. Min Range: -3.5341

Max Act: 3.1130. Min Act: -0.0000

Data Index: 1295288 (The Pile)

Max Activating Token Index: 316

Text #4

Max Range: 3.5341. Min Range: -3.5341

Max Act: 2.9131. Min Act: -0.0000

Data Index: 1417861 (The Pile)

Max Activating Token Index: 297

Text #5

Max Range: 3.5341. Min Range: -3.5341

Max Act: 3.0519. Min Act: -0.0000

Data Index: 273944 (The Pile)

Max Activating Token Index: 249

Text #6

Max Range: 3.5341. Min Range: -3.5341

Max Act: 2.9025. Min Act: -0.0000

Data Index: 264998 (The Pile)

Max Activating Token Index: 272

Text #7

Max Range: 3.5341. Min Range: -3.5341

Max Act: 2.9026. Min Act: -0.0000

Data Index: 918934 (The Pile)

Max Activating Token Index: 158

Text #8

Max Range: 3.5341. Min Range: -3.5341

Max Act: 3.1912. Min Act: -0.0000

Data Index: 249945 (The Pile)

Max Activating Token Index: 663

Text #9

Max Range: 3.5341. Min Range: -3.5341

Max Act: 2.8810. Min Act: -0.0000

Data Index: 1511263 (The Pile)

Max Activating Token Index: 897

Text #10

Max Range: 3.5341. Min Range: -3.5341

Max Act: 2.9901. Min Act: -0.0000

Data Index: 871641 (The Pile)

Max Activating Token Index: 688

Text #11

Max Range: 3.5341. Min Range: -3.5341

Max Act: 2.9906. Min Act: -0.0000

Data Index: 1315232 (The Pile)

Max Activating Token Index: 106

Text #12

Max Range: 3.5341. Min Range: -3.5341

Max Act: 3.0988. Min Act: -0.0000

Data Index: 1086104 (The Pile)

Max Activating Token Index: 657

Text #13

Max Range: 3.5341. Min Range: -3.5341

Max Act: 2.9931. Min Act: -0.0000

Data Index: 622568 (The Pile)

Max Activating Token Index: 123

Text #14

Max Range: 3.5341. Min Range: -3.5341

Max Act: 2.9718. Min Act: -0.0000

Data Index: 1928884 (The Pile)

Max Activating Token Index: 378

Text #15

Max Range: 3.5341. Min Range: -3.5341

Max Act: 3.0031. Min Act: -0.0000

Data Index: 244146 (The Pile)

Max Activating Token Index: 649

Text #16

Max Range: 3.5341. Min Range: -3.5341

Max Act: 2.9597. Min Act: -0.0000

Data Index: 1103838 (The Pile)

Max Activating Token Index: 412

Text #17

Max Range: 3.5341. Min Range: -3.5341

Max Act: 3.0885. Min Act: -0.0000

Data Index: 39612 (The Pile)

Max Activating Token Index: 296

Text #18

Max Range: 3.5341. Min Range: -3.5341

Max Act: 2.7956. Min Act: -0.0000

Data Index: 1765140 (The Pile)

Max Activating Token Index: 148

Text #19

Max Range: 3.5341. Min Range: -3.5341

Max Act: 2.9428. Min Act: -0.0000

Data Index: 1513974 (The Pile)

Max Activating Token Index: 826