Buffered cloned operators in a streaming application

摘要

A streams manager clones a portion of a primary flow graph to a virtual machine with a buffer to assure no data is lost if the corresponding portion of the primary flow graph fails. The buffer can be on the input of the cloned portion or on the output of the cloned portion. Cloning a portion of a primary flow graph with a buffer assures no data is lost when the corresponding portion of the primary flow graph fails. When the primary flow graph recovers from the failure, the processing may be switched back to the primary flow graph, which causes the buffer to begin buffering once again.

主权项

1. An apparatus comprising:
at least one processor; a memory coupled to the at least one processor; a streaming application residing in the memory and executed by the at least one processor, the streaming application comprising a primary flow graph that includes a plurality of operators that process a plurality of data tuples, wherein the primary flow graph comprises a first portion for which data loss needs to be prevented; and a streams manager residing in the memory and executed by the at least one processor, the streams manager cloning the first portion of the primary flow graph to a virtual machine, wherein the cloned first portion comprises a buffer that stores data tuples for the cloned first portion, the streams manager detecting when the first portion of the primary flow graph has a failure, and in response to detecting the failure, notifies the cloned first portion of the failure in the first portion of the primary flow graph, and in response, the cloned first portion outputs data tuples to the primary flow graph to avoid data loss by the first portion of the primary flow graph that had the failure.