Platform for continuous graph update and computation

摘要

A continuous stream data (e.g., messages, tweets) is received by ingest nodes of a platform. The ingest nodes may analyze the data to create a transaction of graph updates, assign a sequence number to the transaction, and distribute the graph updates with the sequence number to graph nodes of the platform. The graph nodes may store graph updates from ingest nodes, and then the ingest nodes may report graph update progresses in a progress table. A snapshot may be taken based on the progress table, and then graph-mining computation may be implemented. Tolerate failures and decaying may be supported and incremental expansion may be allowed to cope with increasing update rates and computation needs.

主权项

1. A computer-implemented method for processing a continuous data stream, the method comprising:
receiving a record of the continuous data stream; generating, by an ingest node, one or more graph-update operations based on the record; assigning a sequence number to the one or more graph-update operations; distributing the one or more graph-update operations with the sequence number to a plurality of graph nodes; recording the sequence number in a global progress table after the one or more graph-update operations are stored by the plurality of graph nodes, the global progress table comprising a vector including the sequence number and another sequence number assigned by another ingest node that generates and distributes one or more other graph-update operations to the plurality of graph nodes; defining an end of an epoch based on the global progress table based on the vector; and producing a graph structure snapshot by executing the one or more graph-update operations that are stored in the plurality of graph nodes and are within the epoch and by executing the one or more other graph-update operations.