Managing metadata for a distributed processing system with manager agents and worker agents

摘要

Methods, systems, and apparatus are described for managing metadata for a distributed processing system. In one aspect, a method includes receiving a computation specification that includes a set of grains that specifies an initial state for a computation that is to be performed by a distributed processing system, wherein each grain comprises metadata that specifies a portion of the initial state for the computation; storing a grain hierarchy that represents a state for the computation based on a grain type associated with each grain, the grain hierarchy comprising subscription grains for subscriptions for the grain hierarchy, each subscription corresponding to one or more grains included in the grain hierarchy, and each subscription specifying one or more actions to be performed by the hub device; and performing, during performance of the computation, at least one of the actions specified by at least one of the subscriptions.

主权项

1. A method implemented by data processing apparatus, the method comprising:
receiving, at a hub device that includes one or more data processing apparatus, a computation specification that includes a set of grains that specifies an initial state for a computation that is to be performed by a distributed processing system that includes a plurality of data processors, wherein each grain included in the set of grains comprises metadata that specifies a portion of the initial state for the computation; storing, by the hub device, a grain hierarchy that represents a state for the computation based on a grain type associated with each grain included in the set of grains, the grain hierarchy comprising subscription grains for one or more subscriptions for the grain hierarchy, each subscription corresponding to one or more grains included in the grain hierarchy, and each subscription specifying one or more actions to be performed by the hub device; and performing, by the hub device and during performance of the computation, at least one of the one or more actions specified by at least one of the one or more subscriptions; wherein the grain hierarchy comprises:
one or more manager agent grains specifying metadata for one or more manager agent processes that manage the performance of the computation by the distributed processing system; andone or more worker agent grains including, for at least one of one or more tasks to be performed by the distributed processing system, metadata for one or more worker agents that each correspond to a subset of the plurality of data processors for performing the task.