Hierarchical chunking of objects in a distributed storage system

摘要

Placement of object replicas in a distributed storage system includes, at a first instance, opening a journal for storage of object chunks. An object is received, which comprises one or more chunks. Each chunk comprises one or more storage blocks. The blocks for a single chunk are stored in a single journal. Global metadata for the object is stored, which includes a list of chunks for the object. Local metadata for the chunk is stored, which includes a block list identifying each block of the plurality of blocks. The local metadata is associated with the journal. The journal is later closed. The journal is subsequently replicated to a second instance. The global metadata is updated to reflect the replication, whereas the local metadata is unchanged by the replication.

主权项

1. A method for managing placement of object replicas in a distributed storage system, comprising:
at a first instance of the distributed storage system, having one or more processors and memory, wherein the memory stores one or more programs for execution by the one or more processors: receiving a first object; splitting the first object into a plurality of object chunks and splitting a first object chunk of the plurality of object chunks into a plurality of blocks; storing the plurality of blocks in a first journal; storing global metadata for the first object, wherein the global metadata includes a list of the plurality of object chunks, and wherein the list includes a respective identifier for each of the object chunks; storing local metadata for the first object chunk, wherein the local metadata includes a block list identifying each block of the plurality of blocks, and wherein the local metadata is associated with the first journal; replicating the first journal to a second instance of the distributed storage system, wherein the global metadata is updated to reflect the replication, whereas the local metadata is unchanged by the replication.