Efficient Data Reads From Distributed Storage Systems

摘要

A method of distributing data in a distributed storage system includes receiving a file into non-transitory memory and dividing the received file into chunks. The chunks are data-chunks and non-data chunks. The method also includes grouping one or more of the data chunks and one or more of the non-data chunks in a group. One or more chunks of the group is capable of being reconstructed from other chunks of the group. The method also includes distributing the chunks of the group to storage devices of the distributed storage system based on a hierarchy of the distributed storage system. The hierarchy includes maintenance domains having active and inactive states, each storage device associated with a maintenance domain, the chunks of a group are distributed across multiple maintenance domains to maintain the ability to reconstruct chunks of the group when a maintenance domain is in an inactive state.

主权项

1. A method of distributing data in a distributed storage system, the method comprising:
receiving, at data processing hardware, a file; dividing, by the data processing hardware, the received file into chunks, the chunks being data-chunks and non-data chunks; grouping, by the data processing hardware, one or more of the data chunks and one or more of the non-data chunks in a group, one or more of the data chunks or one or more of the non-data chunks of the group capable of being reconstructed from other chunks of the group; determining, by the data processing hardware, a distribution of the chunks of the group among storage devices of the distributed storage system based on a maintenance hierarchy of the distributed storage system, the maintenance hierarchy comprising hierarchical maintenance levels and maintenance domains, each maintenance domain having an active state or an inactive state, each maintenance domain spanning one or more adjacent hierarchical maintenance levels, each storage device associated with at least one maintenance domain; and distributing, by the data processing hardware, the chunks of the group to the storage devices based on the determined distribution, the chunks of the group being distributed across multiple maintenance domains to maintain an ability to reconstruct chunks of the group when a maintenance domain is in the inactive state.