Device and method for optimization of data processing in a MapReduce framework

摘要

A map reduce frame work for large scale data processing is optimized by the method of the invention that can be implemented by a master node. The method comprises reception of data from worker nodes on read pointer locations pointing to input data of tasks executed by these worker nodes and stealing of work from these tasks, the work being stolen being applied to input data that have not yet been processed by the task from which work is stolen.