Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 1639 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 3.8907. Min Range: -3.8907

Max Act: 3.8907. Min Act: -0.0000

Data Index: 1107351 (The Pile)

Max Activating Token Index: 217

Text #1

Max Range: 3.8907. Min Range: -3.8907

Max Act: 3.2389. Min Act: -0.0000

Data Index: 1842751 (The Pile)

Max Activating Token Index: 535

Text #2

Max Range: 3.8907. Min Range: -3.8907

Max Act: 3.6799. Min Act: -0.0000

Data Index: 1936270 (The Pile)

Max Activating Token Index: 71

Text #3

Max Range: 3.8907. Min Range: -3.8907

Max Act: 3.4828. Min Act: -0.0000

Data Index: 1300638 (The Pile)

Max Activating Token Index: 242

Text #4

Max Range: 3.8907. Min Range: -3.8907

Max Act: 3.3017. Min Act: -0.0000

Data Index: 757098 (The Pile)

Max Activating Token Index: 948

Text #5

Max Range: 3.8907. Min Range: -3.8907

Max Act: 3.3606. Min Act: -0.0000

Data Index: 1671659 (The Pile)

Max Activating Token Index: 652

Text #6

Max Range: 3.8907. Min Range: -3.8907

Max Act: 3.0088. Min Act: -0.0000

Data Index: 978755 (The Pile)

Max Activating Token Index: 289

Text #7

Max Range: 3.8907. Min Range: -3.8907

Max Act: 3.0202. Min Act: -0.0000

Data Index: 736701 (The Pile)

Max Activating Token Index: 368

Text #8

Max Range: 3.8907. Min Range: -3.8907

Max Act: 3.0225. Min Act: -0.0000

Data Index: 1234362 (The Pile)

Max Activating Token Index: 221

Text #9

Max Range: 3.8907. Min Range: -3.8907

Max Act: 3.4643. Min Act: -0.0000

Data Index: 703408 (The Pile)

Max Activating Token Index: 767

Text #10

Max Range: 3.8907. Min Range: -3.8907

Max Act: 2.9400. Min Act: -0.0000

Data Index: 1958363 (The Pile)

Max Activating Token Index: 624

Text #11

Max Range: 3.8907. Min Range: -3.8907

Max Act: 3.0618. Min Act: -0.0000

Data Index: 734436 (The Pile)

Max Activating Token Index: 707

Text #12

Max Range: 3.8907. Min Range: -3.8907

Max Act: 2.9028. Min Act: -0.0000

Data Index: 1963236 (The Pile)

Max Activating Token Index: 271

Text #13

Max Range: 3.8907. Min Range: -3.8907

Max Act: 3.3017. Min Act: -0.0000

Data Index: 913650 (The Pile)

Max Activating Token Index: 57

Text #14

Max Range: 3.8907. Min Range: -3.8907

Max Act: 3.1621. Min Act: -0.0000

Data Index: 1003886 (The Pile)

Max Activating Token Index: 101

Text #15

Max Range: 3.8907. Min Range: -3.8907

Max Act: 2.8023. Min Act: -0.0000

Data Index: 1692529 (The Pile)

Max Activating Token Index: 347

Text #16

Max Range: 3.8907. Min Range: -3.8907

Max Act: 2.8490. Min Act: -0.0000

Data Index: 658391 (The Pile)

Max Activating Token Index: 667

Text #17

Max Range: 3.8907. Min Range: -3.8907

Max Act: 2.5185. Min Act: -0.0000

Data Index: 65585 (The Pile)

Max Activating Token Index: 718

Text #18

Max Range: 3.8907. Min Range: -3.8907

Max Act: 2.6348. Min Act: -0.0000

Data Index: 1503210 (The Pile)

Max Activating Token Index: 941

Text #19

Max Range: 3.8907. Min Range: -3.8907

Max Act: 2.7491. Min Act: -0.0000

Data Index: 1317283 (The Pile)

Max Activating Token Index: 483