Managing data paths in an operator graph

摘要

Embodiments of the disclosure provide a method and, system for processing data such as a stream of tuples. The method can include receiving the stream of tuples to be processed by a plurality of stream operators operating on one or more computer processors. The method can include creating an overflow path that includes at least one stream operator that performs processing duplicative to at least one stream operator from the plurality of stream operators. The method can include monitoring a stream operator for a triggering condition. The method can include identifying a tuple from the stream of tuples to process on the overflow path. The method can include processing, on the overflow path, the identified tuple from the stream of tuples in response to the presence of the triggering condition.

主权项

1. A system for processing a stream of tuples, each tuple having one or more attributes, comprising:
one or more processing elements operating on one or more hardware compute nodes having one or more computer processors configured to process the attributes of each tuple from the stream of tuples; a stream manager operating on one or more computer processors configured to manage the one or more processing elements, the stream manager is configured to:
identify a tuple from the stream of tuples to process on an overflow path by selecting the tuple for the overflow path based on the presence of a complex attribute of the tuple that is determined through a complexity score, the complexity score is determined by a numerical weighing for a tuple so that a stream operator can identify a predicted processing time, wherein the overflow path includes at least one processing element that performs processing duplicative to another processing element from the one or more processing elements;monitor one or more processing elements for a triggering condition;create, in response to a presence of the triggering condition, the overflow path;process, on the overflow path, an identified tuple from the stream of tuples in response to a presence of the overflow path;monitor at least one processing element on the overflow path for a high traffic limit and a low traffic limit, wherein the high traffic limit comprises a processing rate at a respective processing element that is at least 10% greater than a maximum allowable processing rate, and wherein the low traffic limit comprises a processing rate at a respective processing element that is more than 2% below a minimum allowable processing rate;create an additional overflow path responsive to detecting the high traffic limit; anddeactivate, responsive to detecting the low traffic limit, a respective overflow path that triggers the low traffic limit.