Elastic auto-parallelization for stream processing applications based on measured throughput and congestion

摘要

A method for adjusting a data parallel region of a stream processing application includes measuring congestion of each parallel channel of the data parallel region, measuring a total throughput of all the parallel channels, and adjusting the number of parallel channels based on the current measured congestion and throughput.

主权项

1. A computer program product for adjusting a data parallel region of a stream processing application, the computer program product comprising a non-transitory computer readable storage medium having program code embodied therewith, the program code executable by a processor, to perform method steps comprising:
measuring congestion of each parallel channel of the data parallel region; measuring a total throughput of all the parallel channels; and adjusting the number of parallel channels based on the current measured congestion and throughput, wherein each parallel channel comprises a same at least one software operator that performs an operation on a stream of data received from an input operator, wherein measuring the congestion for a given channel comprises calculating a blocking period from a difference between a first time a send call across the channel provides a notification that the call will block and a second time the send call across the channel provides a notification that room is available.