Modular architecture for extreme-scale distributed processing applications

摘要

Embodiments of the present invention relate to a new data center architecture that provides for efficient processing in distributed analytics applications. In one embodiment, a distributed processing node is provided. The node comprises a plurality of subnodes. Each subnode includes at least one processor core operatively connected to a memory. A first interconnect operatively connects each of the plurality of subnodes within the node. A second interconnect operably connects each of the plurality of subnodes to a storage. A process runs on a first of the plurality of subnodes, the process being operative to retrieve data from the memory of the first subnode. The process interrogates the memory of the first subnode for requested data. If the requested data is not found in the memory of the first subnode, the process interrogates the memory of at least one other subnode of the plurality of subnodes via the first interconnect. If the requested data is found in the memory of the other subnode, the process copies the requested data to the memory of the first subnode. If the requested data is not found in the memory of the first subnode or the memories of at least one subnode of the plurality of subnodes, the process interrogates the storage via the second interconnect.

主权项

1. A system comprising:
a distributed processing node comprising a plurality of subnodes, each subnode including:
at least one processor core operatively connected to a memory;a first interconnect operatively connected to each of the plurality of subnodes;a second interconnect operatively connected to each of the plurality of subnodes and to a storage, the storage comprising a first storage unit and a second storage unit, the second storage unit having lower access time and latency than the first storage unit; a process running on a first of the plurality of subnodes, the process being operative to retrieve data from the memory of the first subnode; wherein:
the process interrogates the memory of the first subnode for requested data;if the requested data is not found in the memory of the first subnode, the process interrogates the memory of at least one other subnode of the plurality of subnodes via the first interconnect;if the requested data is found in the memory of the other subnode, the process copies the requested data to the memory of the first subnode; andif the requested data is not found in the memory of the first subnode or the memory of at least another subnode of the plurality of subnodes, the process interrogates the storage via the second interconnect; a storage manager operative to allocate data between the first and second storage units based on access patterns, the storage manager preferentially relocating non-sequentially accessed data to the second storage unit from the first storage unit.