Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 1561 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 0.5801. Min Range: -0.5801

Max Act: 0.5801. Min Act: -0.0000

Data Index: 610670 (The Pile)

Max Activating Token Index: 250

Text #1

Max Range: 0.5801. Min Range: -0.5801

Max Act: 0.4628. Min Act: -0.0000

Data Index: 573109 (The Pile)

Max Activating Token Index: 625

Text #2

Max Range: 0.5801. Min Range: -0.5801

Max Act: 0.3529. Min Act: -0.0000

Data Index: 362709 (The Pile)

Max Activating Token Index: 282

Text #3

Max Range: 0.5801. Min Range: -0.5801

Max Act: 0.3054. Min Act: -0.0000

Data Index: 34815 (The Pile)

Max Activating Token Index: 472

Text #4

Max Range: 0.5801. Min Range: -0.5801

Max Act: 0.3014. Min Act: -0.0000

Data Index: 534656 (The Pile)

Max Activating Token Index: 545

Text #5

Max Range: 0.5801. Min Range: -0.5801

Max Act: 0.2782. Min Act: -0.0000

Data Index: 489400 (The Pile)

Max Activating Token Index: 703

Text #6

Max Range: 0.5801. Min Range: -0.5801

Max Act: 0.3120. Min Act: -0.0000

Data Index: 831258 (The Pile)

Max Activating Token Index: 488

Text #7

Max Range: 0.5801. Min Range: -0.5801

Max Act: 0.2849. Min Act: -0.0000

Data Index: 1000264 (The Pile)

Max Activating Token Index: 577

Text #8

Max Range: 0.5801. Min Range: -0.5801

Max Act: 0.3118. Min Act: -0.0000

Data Index: 1783653 (The Pile)

Max Activating Token Index: 930

Text #9

Max Range: 0.5801. Min Range: -0.5801

Max Act: 0.3173. Min Act: -0.0000

Data Index: 955439 (The Pile)

Max Activating Token Index: 226

Text #10

Max Range: 0.5801. Min Range: -0.5801

Max Act: 0.2950. Min Act: -0.0000

Data Index: 1317451 (The Pile)

Max Activating Token Index: 459

Text #11

Max Range: 0.5801. Min Range: -0.5801

Max Act: 0.2550. Min Act: -0.0000

Data Index: 299362 (The Pile)

Max Activating Token Index: 484

Text #12

Max Range: 0.5801. Min Range: -0.5801

Max Act: 0.2659. Min Act: -0.0000

Data Index: 364056 (The Pile)

Max Activating Token Index: 338

Text #13

Max Range: 0.5801. Min Range: -0.5801

Max Act: 0.2367. Min Act: -0.0000

Data Index: 1844795 (The Pile)

Max Activating Token Index: 680

Text #14

Max Range: 0.5801. Min Range: -0.5801

Max Act: 0.2488. Min Act: -0.0000

Data Index: 1114034 (The Pile)

Max Activating Token Index: 602

Text #15

Max Range: 0.5801. Min Range: -0.5801

Max Act: 0.2541. Min Act: -0.0000

Data Index: 109887 (The Pile)

Max Activating Token Index: 176

Text #16

Max Range: 0.5801. Min Range: -0.5801

Max Act: 0.2370. Min Act: -0.0000

Data Index: 1425229 (The Pile)

Max Activating Token Index: 330

Text #17

Max Range: 0.5801. Min Range: -0.5801

Max Act: 0.2491. Min Act: -0.0000

Data Index: 79316 (The Pile)

Max Activating Token Index: 183

Text #18

Max Range: 0.5801. Min Range: -0.5801

Max Act: 0.2326. Min Act: -0.0000

Data Index: 334427 (The Pile)

Max Activating Token Index: 171

Text #19

Max Range: 0.5801. Min Range: -0.5801

Max Act: 0.2317. Min Act: -0.0000

Data Index: 458572 (The Pile)

Max Activating Token Index: 181