Data parallelism and parallel operations in stream processing

摘要

A stream processing platform that provides fast execution of stream processing applications within a safe runtime environment. The platform includes a stream compiler that converts a representation of a stream processing application into executable program modules for a safe environment. The platform allows users to specify aspects of the program that contribute to generation of modules that execute as intended. A user may specify aspects to control a type of implementation for loops, order of execution for parallel paths, whether multiple instances of an operation can be performed in parallel or whether certain operations should be executed in separate threads. In addition, the stream compiler may generate executable modules in a way that cause a safe runtime environment to allocate memory or otherwise operate efficiently.

主权项

1. A method of generating machine-executable instructions for processing at least one stream of data values, the method comprising:
receiving:
an indication of at least one first operation to be performed on the at least one stream, the at least one stream is defined by a schema and comprises a plurality of tuples; andan indication of at least one second operation to be performed on the at least one stream; generating a first set of machine-executable instructions to perform the at least one first operation; and generating a second set of machine-executable instructions to perform the at least one second operation, wherein:
the first set and the second set are generated based on user input such that, when the user input comprises an indication that the at least one second operation is to execute in a separate thread, the first set and the second set are encoded to, when executed, be separated by a thread boundary and, when the user input does not comprise the indication, the first set and the second set are encoded to be executed in a same thread.