Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 75 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 0.9326. Min Range: -0.9326

Max Act: 0.9326. Min Act: -0.0000

Data Index: 1381155 (The Pile)

Max Activating Token Index: 794

Text #1

Max Range: 0.9326. Min Range: -0.9326

Max Act: 0.7873. Min Act: -0.0000

Data Index: 1302331 (The Pile)

Max Activating Token Index: 898

Text #2

Max Range: 0.9326. Min Range: -0.9326

Max Act: 0.7214. Min Act: -0.0000

Data Index: 1373251 (The Pile)

Max Activating Token Index: 938

Text #3

Max Range: 0.9326. Min Range: -0.9326

Max Act: 0.6688. Min Act: -0.0000

Data Index: 1073502 (The Pile)

Max Activating Token Index: 95

Text #4

Max Range: 0.9326. Min Range: -0.9326

Max Act: 0.6562. Min Act: -0.0000

Data Index: 473250 (The Pile)

Max Activating Token Index: 607

Text #5

Max Range: 0.9326. Min Range: -0.9326

Max Act: 0.7022. Min Act: -0.0000

Data Index: 463569 (The Pile)

Max Activating Token Index: 958

Text #6

Max Range: 0.9326. Min Range: -0.9326

Max Act: 0.7100. Min Act: -0.0000

Data Index: 1684418 (The Pile)

Max Activating Token Index: 218

Text #7

Max Range: 0.9326. Min Range: -0.9326

Max Act: 0.6597. Min Act: -0.0000

Data Index: 296405 (The Pile)

Max Activating Token Index: 202

Text #8

Max Range: 0.9326. Min Range: -0.9326

Max Act: 0.7313. Min Act: -0.0000

Data Index: 1685989 (The Pile)

Max Activating Token Index: 841

Text #9

Max Range: 0.9326. Min Range: -0.9326

Max Act: 0.6682. Min Act: -0.0000

Data Index: 247535 (The Pile)

Max Activating Token Index: 22

Text #10

Max Range: 0.9326. Min Range: -0.9326

Max Act: 0.6222. Min Act: -0.0000

Data Index: 1219488 (The Pile)

Max Activating Token Index: 874

Text #11

Max Range: 0.9326. Min Range: -0.9326

Max Act: 0.6265. Min Act: -0.0000

Data Index: 1486314 (The Pile)

Max Activating Token Index: 308

Text #12

Max Range: 0.9326. Min Range: -0.9326

Max Act: 0.6177. Min Act: -0.0000

Data Index: 187428 (The Pile)

Max Activating Token Index: 617

Text #13

Max Range: 0.9326. Min Range: -0.9326

Max Act: 0.5776. Min Act: -0.0000

Data Index: 483590 (The Pile)

Max Activating Token Index: 517

Text #14

Max Range: 0.9326. Min Range: -0.9326

Max Act: 0.5822. Min Act: -0.0000

Data Index: 1367941 (The Pile)

Max Activating Token Index: 527

Text #15

Max Range: 0.9326. Min Range: -0.9326

Max Act: 0.5948. Min Act: -0.0000

Data Index: 595525 (The Pile)

Max Activating Token Index: 794

Text #16

Max Range: 0.9326. Min Range: -0.9326

Max Act: 0.5905. Min Act: -0.0000

Data Index: 709704 (The Pile)

Max Activating Token Index: 180

Text #17

Max Range: 0.9326. Min Range: -0.9326

Max Act: 0.5451. Min Act: -0.0000

Data Index: 959876 (The Pile)

Max Activating Token Index: 667

Text #18

Max Range: 0.9326. Min Range: -0.9326

Max Act: 0.5630. Min Act: -0.0000

Data Index: 816198 (The Pile)

Max Activating Token Index: 436

Text #19

Max Range: 0.9326. Min Range: -0.9326

Max Act: 0.5493. Min Act: -0.0000

Data Index: 1407188 (The Pile)

Max Activating Token Index: 913