Buffered cloned operators in a streaming application

摘要

A streams manager clones a portion of a primary flow graph to a virtual machine with a buffer to assure no data is lost if the corresponding portion of the primary flow graph fails. The buffer can be on the input of the cloned portion or on the output of the cloned portion. Cloning a portion of a primary flow graph with a buffer assures no data is lost when the corresponding portion of the primary flow graph fails. When the primary flow graph recovers from the failure, the processing may be switched back to the primary flow graph, which causes the buffer to begin buffering once again.

主权项

1. A computer-implemented method executed by at least one processor for managing a streaming application, the method comprising:
executing a streaming application that comprises a primary flow graph that includes a plurality of operators that process a plurality of data tuples, wherein the primary flow graph comprises a first portion for which data loss needs to be prevented; cloning the first portion of the primary flow graph to a virtual machine, wherein the cloned first portion comprises a buffer that stores data tuples for the cloned first portion; detecting when the first portion of the primary flow graph has a failure; notifying the cloned first portion of the failure; and in response to the notification of the failure, the cloned first portion outputs data tuples to the primary flow graph to avoid data loss by the first portion of the primary flow graph that had the failure.