Scheduling and execution of DAG-structured computation on RDMA-connected clusters

摘要

A server and/or a client stores a metadata hash map that includes one or more entries associated with keys for data records stored in a cache on a server, wherein the data records comprise a directed acyclic graph (DAG), and the directed acyclic graph is comprised of a collection of one or more nodes connected by one or more edges, each of the nodes representing one or more tasks ordered into a sequence, and each of the edges representing one or more constraints on the nodes connected by the edges. Each of the entries stores metadata for a corresponding data record, wherein the metadata comprises a server-side remote pointer that references the corresponding data record stored in the cache. A selected data record is accessed using a provided key by: (1) identifying potentially matching entries in the metadata hash map using the provided key; (2) accessing data records stored in the cache using the server-side remote pointers from the potentially matching entries; and (3) determining whether the accessed data records match the selected data record using the provided key.

主权项

1. A method for performing a sequence of tasks, comprising:
storing one or more data records comprising a directed acyclic graph (DAG) in a server, wherein the directed acyclic graph is comprised of a collection of one or more nodes connected by one or more edges, each of the nodes represents one or more tasks ordered into a sequence, and each of the edges connected to a node represents one or more constraints on the task represented by the node; wherein the server stores a metadata hash map that includes one or more entries associated with each of the data records, each of the entries is associated with a key for a corresponding data record, each of the entries store metadata for the corresponding data record, and the metadata comprises a server-side remote pointer that references the corresponding data record; wherein one or more clients store at least a subset of the metadata hash map, the clients identify one or more of the server-side remote pointers in the subset of the metadata hash map using a provided key, the clients perform a remote direct memory access (RDMA) operation using the identified server-side remote pointers to access the data records stored in the server, and the clients execute the tasks associated with the accessed data records using the constraints associated with the accessed data records.