Archival data flow management

摘要

Methods and systems are provided herein to allow efficient management of data flowing in and out of an archival data storage system. In an embodiment, storage entities keep very little state information in memory to provide higher throughput. Further, storage entities may send data in large chunks to facilitate high throughput. Techniques such as batching and coalescing may be used by various storage entities to provide efficiency.

主权项

1. A computer system for managing data storage, comprising:
a plurality of data storage devices; a plurality of storage nodes, each of the plurality of data storage nodes being operably connected to one or more of the plurality of data storage devices; a storage node manager operably connected to the plurality of data storage nodes; one or more processors; memory, including executable instructions that, when executed by the one or more processors, cause the one or more processors to collectively at least:
receive, by the storage node manager, a request to store a data object;obtain, by the storage node manager, the data object to be stored; andallocate, by the storage node manager, storage space to store a plurality of redundantly encoded data components that are generated using at least one erasure code and based at least in part on the data object to be stored; for each encoded data component of the plurality of redundantly encoded data components:
generate a data collection that includes at least the encoded data component and redundantly encoded data components associated with another data object, such that the data collection is redundantly encoded; andprovide, by the storage node manager, the encoded data component to a storage node; after providing the encoded data component, make available storage space allocated to the plurality of encoded data components regardless of whether a response is received from the storage node; receive, by a storage node, one of the plurality of encoded data components; and store, by the storage node, the data collection on a storage device connected to the storage node.