PARALLEL PROCESSING OF DATA

摘要

An untrusted application is received at a data center including one or more processing modules and providing a native processing environment. The untrusted application includes a data parallel pipeline. Secured processing environments are used to execute the untrusted application. A data parallel pipeline may specify multiple parallel data objects and parallel operations. Based on the data parallel pipeline, a dataflow graph of deferred parallel data objects and deferred parallel operations may be generated and one or more graph transformations may be applied to the dataflow graph to generate a revised dataflow graph that includes one or more of the deferred parallel data objects and deferred, combined parallel data operations. The deferred, combined parallel operations may be executed to produce materialized parallel data objects corresponding to the deferred parallel data objects.