VOLUME-LEVEL REDUNDANCY CODING TECHNIQUES FOR SEQUENTIAL TRANSFER OPTIMIZED STORAGE DEVICES

摘要

Techniques described and suggested herein include systems and methods for storing, indexing, and retrieving original data of data archives on data storage systems using redundancy coding techniques. For example, redundancy codes, such as erasure codes, may be applied to archives (such as those received from a customer of a computing resource service provider) so as allow the storage of original data of the individual archives available on a minimum of volumes, such as those of a data storage system, while retaining availability, durability, and other guarantees imparted by the application of the redundancy code. Sparse indexing techniques may be implemented so as to reduce the footprint of indexes used to locate the original data, once stored.

主权项

1. A computer-implemented method, comprising:
under the control of one or more computer systems configured with executable instructions,
processing a plurality of archives to be stored on a plurality of volumes so as to:
sort the plurality of archives according to at least one criterion shared by the plurality of archives; anddetermine which archives of the sorted plurality of archives will be stored on each volume of the plurality of volumes;generating indexes for the plurality of volumes, each index of the indexes reflecting the a subset of the sorted plurality of archives to be stored on a respective volume of the plurality of volumes;store the sorted plurality of archives and the generated indexes on a subset of the plurality of volumes, thereby generating a plurality of shards;applying a redundancy code to the sorted plurality of archives and the generated indexes to generate encoded shards; andstoring the encoded shards on corresponding volumes outside the subset of the plurality of volumes.