Network-independent programming model for online processing in distributed systems

摘要

Systems and methods include a distributed system that uses a declarative language to separate application logic from the network technology used to communicate. For example, in a system with a plurality of processing nodes, each processing node may include at least one processor, a networking layer for communicating with other processing nodes, and a memory storing instructions that, when executed by the at least one processor, cause the processing node to perform operations. The operations may include receiving a processing plan using the networking layer and analyzing the processing plan to determine a service to perform and a number of inputs anticipated for the service. The operations may also include performing the determined service. The processing plan defines a processing tree comprising a subset of the plurality of processing nodes, the processing tree having unidirectional communication from the root to a final level of the tree.

主权项

1. A system comprising:
a memory storing a plurality of processing plans, each processing plan including of a series of operations to be performed, each operation including:
a service, andan indication of at least one of a plurality of processing nodes for performing the service,wherein at least one operation of the series of operations, also includes an indication of one or more inputs representing an output of another operation of the series of operations; a first node of the plurality of nodes, wherein the first node includes at least one processor and a memory storing instructions that, when executed by the at least one processor, cause the first node to:
select a particular processing plan from the memory, anddistribute the particular processing plan to second nodes of the plurality of processing nodes, the second nodes being identified in at least one first operation of the particular processing plan, the first operation having no indication of a previous operation; and the plurality of processing nodes, each processing node including at least one processor and a memory storing instructions that, when executed by the at least one processor, cause the processing node to:
receive a processing plan of the plurality of processing plans,determine an anticipated number of inputs, each input being identified in the processing plan as an output from a previous operation,determine whether the anticipated number of inputs has been received, andperform the service indicated in the first operation of the received processing plan when it is determined that the anticipated number of inputs have been received.