System and method for compiler assisted parallelization of a stream processing operator

摘要

A method of enabling compiler assisted parallelization of one or more stream processing operators in a stream processing application, which consists of a data flow graph with operators as vertices connected by streams. The method includes specifying a parallelized version of one or more of the operators, with a parameterized degree of parallelism, in the stream application, evaluating whether or not to use the parallelized operator, deciding the degree of parallelism of the parallelized operator, if there is a need for a parallelized operator.

主权项

1. A method of enabling compiler assisted parallelization of one or more stream processing operators in a stream processing application operating on a computer processing device, the method comprising:
creating a theoretical model of one or more operators in a compiler, wherein the theoretical model is based on a plurality of factors comprising an average processing time per tuple, an average number of accesses to a shared state per tuple, and a time to read/write to the shared state; specifying a parallelized version of the one or more operators in the stream processing application based on the theoretical model; evaluating whether the theoretical model improves the performance of the one or more operators, wherein evaluating whether the theoretical model improves the performance of the one or more operators comprises:
determining the greater of the throughput of a single operator and the throughput of the parallelized operator at compile time; computing a degree of parallelism if the theoretical model improves the performance of the one or more operators; and replacing, by the compiler, the one or more operators with the parallelized version of the one or more operators, in response to computing the degree of parallelism.