Systems and methods for reliably storing data using liquid distributed storage

摘要

Embodiments provide methodologies for reliably storing data within a storage system using liquid distributed storage control. Such liquid distributed storage control operates to compress repair bandwidth utilized within a storage system for data repair processing to the point of operating in a liquid regime. Liquid distributed storage control logic of embodiments may employ a lazy repair policy, repair bandwidth control, a large erasure code, and/or a repair queue. Embodiments of liquid distributed storage control logic may additionally or alternatively implement a data organization adapted to allow the repair policy to avoid handling large objects, instead streaming data into the storage nodes at a very fine granularity.

主权项

1. A method for repair of source data comprising one or more source objects stored as multiple fragments distributed across multiple storage nodes of a storage system, wherein one or more fragments of the multiple fragments includes redundant data for the one or more source objects, the method comprising:
determining that at least one fragment of the multiple fragments is missing from the storage system for a source object of the one or more source objects for which there is no corresponding object instance in a repair queue; adding a corresponding object instance to the repair queue for the source object, wherein the repair queue includes object instances for a plurality of source objects having at least one fragment missing from the storage system; and performing repair processing according to a lazy repair policy, wherein repair operation according to the lazy repair policy allows object instances to accumulate in the repair queue for performing repairs at an average repair rate, R, wherein the average repair rate, R, is selected such that the performing repairs at the average repair rate, R, results in processing source objects associated with the queued object instances to complete before a loss rate of fragments results in fewer than k fragments being available in the storage system for any source object of the one or more source objects, wherein k is a number of source fragments per source object.