Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 97 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 7.3162. Min Range: -7.3162

Max Act: 7.3162. Min Act: -0.0000

Data Index: 1462907 (The Pile)

Max Activating Token Index: 277

Text #1

Max Range: 7.3162. Min Range: -7.3162

Max Act: 6.2044. Min Act: -0.0000

Data Index: 382376 (The Pile)

Max Activating Token Index: 141

Text #2

Max Range: 7.3162. Min Range: -7.3162

Max Act: 6.5797. Min Act: -0.0000

Data Index: 386194 (The Pile)

Max Activating Token Index: 185

Text #3

Max Range: 7.3162. Min Range: -7.3162

Max Act: 6.1875. Min Act: -0.0000

Data Index: 1747026 (The Pile)

Max Activating Token Index: 322

Text #4

Max Range: 7.3162. Min Range: -7.3162

Max Act: 5.7678. Min Act: -0.0000

Data Index: 1284103 (The Pile)

Max Activating Token Index: 263

Text #5

Max Range: 7.3162. Min Range: -7.3162

Max Act: 5.6914. Min Act: -0.0000

Data Index: 1607022 (The Pile)

Max Activating Token Index: 611

Text #6

Max Range: 7.3162. Min Range: -7.3162

Max Act: 5.6110. Min Act: -0.0000

Data Index: 583974 (The Pile)

Max Activating Token Index: 359

Text #7

Max Range: 7.3162. Min Range: -7.3162

Max Act: 5.7552. Min Act: -0.0000

Data Index: 1570437 (The Pile)

Max Activating Token Index: 110

Text #8

Max Range: 7.3162. Min Range: -7.3162

Max Act: 5.3172. Min Act: -0.0000

Data Index: 1699479 (The Pile)

Max Activating Token Index: 550

Text #9

Max Range: 7.3162. Min Range: -7.3162

Max Act: 5.7880. Min Act: -0.0000

Data Index: 1554977 (The Pile)

Max Activating Token Index: 98

Text #10

Max Range: 7.3162. Min Range: -7.3162

Max Act: 5.4112. Min Act: -0.0000

Data Index: 1124654 (The Pile)

Max Activating Token Index: 49

Text #11

Max Range: 7.3162. Min Range: -7.3162

Max Act: 5.3809. Min Act: -0.0000

Data Index: 1187335 (The Pile)

Max Activating Token Index: 259

Text #12

Max Range: 7.3162. Min Range: -7.3162

Max Act: 5.3656. Min Act: -0.0000

Data Index: 198175 (The Pile)

Max Activating Token Index: 94

Text #13

Max Range: 7.3162. Min Range: -7.3162

Max Act: 5.2437. Min Act: -0.0000

Data Index: 316151 (The Pile)

Max Activating Token Index: 387

Text #14

Max Range: 7.3162. Min Range: -7.3162

Max Act: 4.8737. Min Act: -0.0000

Data Index: 1037030 (The Pile)

Max Activating Token Index: 19

Text #15

Max Range: 7.3162. Min Range: -7.3162

Max Act: 4.9246. Min Act: -0.0000

Data Index: 1753082 (The Pile)

Max Activating Token Index: 861

Text #16

Max Range: 7.3162. Min Range: -7.3162

Max Act: 5.1130. Min Act: -0.0000

Data Index: 1169566 (The Pile)

Max Activating Token Index: 120

Text #17

Max Range: 7.3162. Min Range: -7.3162

Max Act: 4.7553. Min Act: -0.0000

Data Index: 1690580 (The Pile)

Max Activating Token Index: 102

Text #18

Max Range: 7.3162. Min Range: -7.3162

Max Act: 4.8595. Min Act: -0.0000

Data Index: 439773 (The Pile)

Max Activating Token Index: 615

Text #19

Max Range: 7.3162. Min Range: -7.3162

Max Act: 4.6988. Min Act: -0.0000

Data Index: 371391 (The Pile)

Max Activating Token Index: 539