System and method for distributing and accessing files in a distributed storage system

摘要

A system and method for distributing and accessing files in a distributed storage system uses an ordered list of the storage nodes in the system to determine the storage node on which a file is stored. The distributed storage system includes a cluster of storage nodes and may also include one or more client nodes that participate in the system as storage resources. Each node (client and storage) stores an ordered list of the storage nodes in the system, allowing any of the nodes to access the file. The list is updated whenever a new storage node is added to the system, an existing storage node is removed from the system, or a new storage node is swapped with an existing storage node. Each one of the nodes may independently compute a new mapping of files to the storage nodes when the ordered list is changed.

主权项

1. A computer-implemented method to distribute files across multiple heterogeneous storage nodes without using a centralized server in a distributed storage system, the method comprising:
computing a first mapping of buckets to partitions using a first deterministic function applied to an ordered list of the storage nodes, wherein each bucket stores at least a portion of the files, and at least one of the partitions is fragmented in bucket space, wherein a number of buckets mapped to each partition is based on a ratio of a number of existing buckets to a number of existing partitions in the distributed storage system; computing, using a second deterministic function applied to the ordered list of the storage nodes, a second mapping of the partitions to the multiple storage nodes, wherein each partition is mapped to two or more of the multiple storage nodes based on a predefined redundancy value; combining the first mapping with the second mapping to produce a third mapping of the buckets to the multiple storage nodes; and storing the files across the multiple storage nodes in the distributed storage system according to the third mapping, wherein the ordered list of storage nodes is stored by each of: (i) a plurality of client nodes, and (ii) each of the multiple storage nodes.