Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 3764 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 3.6499. Min Range: -3.6499

Max Act: 3.6499. Min Act: -0.0000

Data Index: 1647100 (The Pile)

Max Activating Token Index: 835

Text #1

Max Range: 3.6499. Min Range: -3.6499

Max Act: 3.4456. Min Act: -0.0000

Data Index: 416185 (The Pile)

Max Activating Token Index: 914

Text #2

Max Range: 3.6499. Min Range: -3.6499

Max Act: 3.0584. Min Act: -0.0000

Data Index: 940388 (The Pile)

Max Activating Token Index: 931

Text #3

Max Range: 3.6499. Min Range: -3.6499

Max Act: 2.8505. Min Act: -0.0000

Data Index: 759051 (The Pile)

Max Activating Token Index: 977

Text #4

Max Range: 3.6499. Min Range: -3.6499

Max Act: 3.1314. Min Act: -0.0000

Data Index: 189687 (The Pile)

Max Activating Token Index: 219

Text #5

Max Range: 3.6499. Min Range: -3.6499

Max Act: 2.9170. Min Act: -0.0000

Data Index: 564361 (The Pile)

Max Activating Token Index: 278

Text #6

Max Range: 3.6499. Min Range: -3.6499

Max Act: 2.7726. Min Act: -0.0000

Data Index: 1911892 (The Pile)

Max Activating Token Index: 460

Text #7

Max Range: 3.6499. Min Range: -3.6499

Max Act: 2.6944. Min Act: -0.0000

Data Index: 1325013 (The Pile)

Max Activating Token Index: 687

Text #8

Max Range: 3.6499. Min Range: -3.6499

Max Act: 2.8406. Min Act: -0.0000

Data Index: 1644030 (The Pile)

Max Activating Token Index: 740

Text #9

Max Range: 3.6499. Min Range: -3.6499

Max Act: 2.6480. Min Act: -0.0000

Data Index: 236841 (The Pile)

Max Activating Token Index: 167

Text #10

Max Range: 3.6499. Min Range: -3.6499

Max Act: 2.6348. Min Act: -0.0000

Data Index: 1274270 (The Pile)

Max Activating Token Index: 921

Text #11

Max Range: 3.6499. Min Range: -3.6499

Max Act: 2.4514. Min Act: -0.0000

Data Index: 863300 (The Pile)

Max Activating Token Index: 583

Text #12

Max Range: 3.6499. Min Range: -3.6499

Max Act: 2.6394. Min Act: -0.0000

Data Index: 1624530 (The Pile)

Max Activating Token Index: 692

Text #13

Max Range: 3.6499. Min Range: -3.6499

Max Act: 2.4890. Min Act: -0.0000

Data Index: 240513 (The Pile)

Max Activating Token Index: 514

Text #14

Max Range: 3.6499. Min Range: -3.6499

Max Act: 2.4588. Min Act: -0.0000

Data Index: 343367 (The Pile)

Max Activating Token Index: 67

Text #15

Max Range: 3.6499. Min Range: -3.6499

Max Act: 2.4567. Min Act: -0.0000

Data Index: 414864 (The Pile)

Max Activating Token Index: 535

Text #16

Max Range: 3.6499. Min Range: -3.6499

Max Act: 2.4710. Min Act: -0.0000

Data Index: 305421 (The Pile)

Max Activating Token Index: 788

Text #17

Max Range: 3.6499. Min Range: -3.6499

Max Act: 2.4394. Min Act: -0.0000

Data Index: 1212359 (The Pile)

Max Activating Token Index: 267

Text #18

Max Range: 3.6499. Min Range: -3.6499

Max Act: 2.4579. Min Act: -0.0000

Data Index: 981321 (The Pile)

Max Activating Token Index: 125

Text #19

Max Range: 3.6499. Min Range: -3.6499

Max Act: 2.4343. Min Act: -0.0000

Data Index: 782000 (The Pile)

Max Activating Token Index: 154