A Method of Processing Multiple Components of a Processing Job in Parallel Across a Plurality of Clusters

摘要

This invention relates to a method of processing multiple components of a processing job in parallel across a plurality of clusters. Data processing clusters are used for computationally expensive jobs to speed up the processing time of those jobs. There is a problem with the known methods in that they are limited to the available resources of a single cluster and they usually do not utilise the available resources optimally resulting in waste and inefficiencies. The method according to the invention partitions a data processing job, allocates the job components across several clusters before creating ProcessJobs for each of the clusters. A plurality of processing server daemons, one on each head node of a cluster receives the ProcessJob and generates a job description for a job manager on the head node. The job is processed across multiple clusters and all the clusters return their results to the processing server via the processing server daemon.