Failure-decoupled volume-level redundancy coding techniques

摘要

Techniques described and suggested herein include systems and methods for storing, indexing, and retrieving original data of data archives on data storage systems using redundancy coding techniques. For example, redundancy codes, such as erasure codes, may be applied to archives (such as those received from a customer of a computing resource service provider) so as allow the storage of original data of the individual archives available on a minimum of volumes, such as those of a data storage system, while retaining availability, durability, and other guarantees imparted by the application of the redundancy code. Sparse indexing techniques may be implemented so as to reduce the footprint of indexes used to locate the original data, once stored. The volumes may be apportioned into failure-decorrelated subsets, and archives stored thereto may be apportioned to such subsets.

主权项

1. A computer-implemented method, comprising:
under the control of one or more computer systems configured with executable instructions,
determining at least two failure-decorrelated subsets of a set of volumes such that:
each failure-decorrelated subset is capable of storing redundancy coded archives; andfailure of a first portion of one or more of the failure-decorrelated subsets, the first portion having at least one common member with a second portion of the one or more of the failure-decorrelated subsets that includes at least one member that does not exist in the first portion, does not affect data integrity of redundancy coded archives stored on at least the second portion of the one or more failure-decorrelated subsets;processing archives to be stored on the set of volumes so as to determine, based on one or more characteristics of the archives, which of the failure-decorrelated subsets to commit the archives;storing the processed archives on a first subset of volumes of the determined failure-decorrelated subset, the first subset of volumes having a number of members corresponding to a quorum quantity of a redundancy code to be applied to the processed archives;applying the redundancy code to the processed archives to generate encoded shards; andstoring the encoded shards on a second subset of volumes of the determined failure-decorrelated subset, the second subset being outside the first subset.