Model: SoLU Model: 8 Layers, 4096 Neurons per Layer

Dataset: The Pile

Neuron 3407 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-8l-pile')

Text #0

Max Range: 0.6584. Min Range: -0.6584

Max Act: 0.6497. Min Act: -0.0000

Data Index: 324033 (The Pile)

Max Activating Token Index: 856

Text #1

Max Range: 0.6584. Min Range: -0.6584

Max Act: 0.6345. Min Act: -0.0000

Data Index: 287302 (The Pile)

Max Activating Token Index: 911

Text #2

Max Range: 0.6584. Min Range: -0.6584

Max Act: 0.6172. Min Act: -0.0000

Data Index: 1415638 (The Pile)

Max Activating Token Index: 394

Text #3

Max Range: 0.6584. Min Range: -0.6584

Max Act: 0.5765. Min Act: -0.0000

Data Index: 796525 (The Pile)

Max Activating Token Index: 648

Text #4

Max Range: 0.6584. Min Range: -0.6584

Max Act: 0.6387. Min Act: -0.0000

Data Index: 278463 (The Pile)

Max Activating Token Index: 323

Text #5

Max Range: 0.6584. Min Range: -0.6584

Max Act: 0.6584. Min Act: -0.0000

Data Index: 1807312 (The Pile)

Max Activating Token Index: 318

Text #6

Max Range: 0.6584. Min Range: -0.6584

Max Act: 0.6409. Min Act: -0.0000

Data Index: 758509 (The Pile)

Max Activating Token Index: 520

Text #7

Max Range: 0.6584. Min Range: -0.6584

Max Act: 0.5751. Min Act: -0.0000

Data Index: 1650363 (The Pile)

Max Activating Token Index: 880

Text #8

Max Range: 0.6584. Min Range: -0.6584

Max Act: 0.5889. Min Act: -0.0000

Data Index: 1845142 (The Pile)

Max Activating Token Index: 340

Text #9

Max Range: 0.6584. Min Range: -0.6584

Max Act: 0.6256. Min Act: -0.0000

Data Index: 1778634 (The Pile)

Max Activating Token Index: 788

Text #10

Max Range: 0.6584. Min Range: -0.6584

Max Act: 0.6190. Min Act: -0.0000

Data Index: 1355163 (The Pile)

Max Activating Token Index: 31

Text #11

Max Range: 0.6584. Min Range: -0.6584

Max Act: 0.5624. Min Act: -0.0000

Data Index: 1172504 (The Pile)

Max Activating Token Index: 511

Text #12

Max Range: 0.6584. Min Range: -0.6584

Max Act: 0.5761. Min Act: -0.0000

Data Index: 709010 (The Pile)

Max Activating Token Index: 584

Text #13

Max Range: 0.6584. Min Range: -0.6584

Max Act: 0.5719. Min Act: -0.0000

Data Index: 514946 (The Pile)

Max Activating Token Index: 44

Text #14

Max Range: 0.6584. Min Range: -0.6584

Max Act: 0.6268. Min Act: -0.0000

Data Index: 1185117 (The Pile)

Max Activating Token Index: 559

Text #15

Max Range: 0.6584. Min Range: -0.6584

Max Act: 0.5994. Min Act: -0.0000

Data Index: 35084 (The Pile)

Max Activating Token Index: 429

Text #16

Max Range: 0.6584. Min Range: -0.6584

Max Act: 0.5646. Min Act: -0.0000

Data Index: 662724 (The Pile)

Max Activating Token Index: 754

Text #17

Max Range: 0.6584. Min Range: -0.6584

Max Act: 0.6043. Min Act: -0.0000

Data Index: 389644 (The Pile)

Max Activating Token Index: 618

Text #18

Max Range: 0.6584. Min Range: -0.6584

Max Act: 0.6181. Min Act: 0.0000

Data Index: 405761 (The Pile)

Max Activating Token Index: 727

Text #19

Max Range: 0.6584. Min Range: -0.6584

Max Act: 0.5577. Min Act: 0.0000

Data Index: 1665841 (The Pile)

Max Activating Token Index: 772