Object store architecture for distributed data processing system

摘要

Embodiments described herein provide an object store that efficiently manages and services objects for use by clients of a distributed data processing system. Illustratively, the object store may be embodied as a quasi-shared storage system that interacts with nodes of the distributed data processing system to service the objects as blocks of data stored on a plurality of storage devices, such as disks, of the storage system. To that end, an architecture of the object store may include an on-disk layout, e.g., of the storage system, and an incore layout, e.g., of the nodes, that cooperate to illustratively convert the blocks to objects for access by the clients.

主权项

1. A method comprising:
mapping each compute node of a plurality of compute nodes to a different volume of a plurality of volumes that defines a logical arrangement of storage space of a storage system, wherein each compute node has read-write access to the volume to which the compute node is mapped and read-only access to those of the plurality of volumes not mapped to the compute node; and in response to receiving an identifier for a data object at a first compute node of the plurality of compute nodes,
identifying an object descriptor associated with the identifier and a plurality of chunk descriptors referenced by the object descriptor, wherein the object descriptor and the plurality of chunk descriptors are arranged on the first compute node according to an incore layout;utilizing the plurality of chunk descriptors to retrieve, via a network from the plurality of volumes, blocks of data stored according to an on-disk layout of the plurality of volumes, wherein the blocks of data include a plurality of chunks described by the plurality of chunk descriptors;the first compute node assembling the plurality of chunks described by the plurality of chunk descriptors into the data object; andreturning the assembled data object.