Unfusing operators from a streaming processing element to avoid congestion

摘要

A streams manager determines when congestion is happening or is predicted to happen in a streaming application, and in response, unfuses one or more operators in the streaming application from its processing element. The selection of which operators to unfuse in which processing elements is made using suitable unfuse criteria, which may include selecting operators that do not maintain state, selecting operators that have a threaded port or queue preceding the operator, selecting operators at natural boundaries, and inserting queues before operators to be unfused. Once one or more operators are unfused, the streams manager can take action to enhance performance of the unfused operator(s). For example, the streams manager can allocate additional resources to the unfused operators, can create clones of unfused operators that process tuples in parallel, can move one or more unfused operators to a public cloud, etc.

主权项

1. An apparatus comprising:
at least one processor; a memory coupled to the at least one processor; a streaming application residing in the memory and executed by the at least one processor, the streaming application comprising a flow graph that includes a plurality of processing elements that process a plurality of data tuples, wherein each of the plurality of processing elements comprises at least one operator; a performance monitor residing in the memory and executed by the at least one processor, the performance monitor monitoring performance of the plurality of processing elements in the flow graph, the performance monitor comprising a congestion detection mechanism that detects when congestion occurs in the plurality of processing elements; and a streams manager residing in the memory and executed by the at least one processor, the streams manager managing the streaming application, and when the performance monitor detects that congestion is occurring in the plurality of processing elements, in response, the streams manager selects based on at least one unfuse criterion at least one operator in the flow graph and unfuses the selected at least one operator from its corresponding processing element to improve congestion, wherein the at least one unfuse criterion specifies to unfuse operators that do not maintain state data.