Compressing tuples in a streaming application

摘要

A method, system, and computer program product to process data in a streaming application are disclosed. The method, system, and computer program product may include receiving a stream of tuples to be processed by a plurality of processing elements operating on a plurality of compute nodes. The method, system, and computer program product may determine whether a first processing element has additional processing capacity. In some embodiments, the method, system, and computer program product determine whether a second processing element, which receives its input from the first processing element, also has additional processing capacity. The method, system, and computer program product may employ compression at the first processing element if one of the first and the second processing element has additional processing capacity.

主权项

1. A system for processing a stream of tuples, comprising:
a plurality of processing elements configured to receive a stream of tuples, the plurality of processing elements operating on two or more compute nodes, each compute node having one or more computer processors, the two or more compute nodes communicatively coupled to each other using one or more communication networks; a performance monitor configured to maintain one or more performance statistics corresponding to each of the plurality of processing elements, the one or more performance statistics indicating whether each of the plurality of processing elements has an additional processing capacity, wherein:
a first of the plurality of processing elements provides an input to a second of the plurality of processing elements, anda third of the plurality of processing elements provides an input to the first processing element; and a stream manager to:
determine whether a bandwidth of a communication network is low, andconfigure the first processing element to employ compression on the stream of tuples if the first processing element has a first additional processing capacity, the second processing element has a second additional processing capacity, and the bandwidth of the communication network is low.