DEVICE AND METHOD FOR OPTIMIZATION OF DATA PROCESSING IN A MapReduce FRAMEWORK

摘要

A map reduce frame work for large scale data processing is optimized by the method of the invention that can be implemented by a master node. The method comprises reception of data from worker nodes on read pointer locations pointing to input data of tasks executed by these worker nodes and stealing of work from these tasks, the work being stolen being applied to input data that have not yet been processed by the task from which work is stolen.

主权项

1. A method for processing data in a map reduce framework, wherein the method is executed by a master node and comprises:
splitting of input data into input data segments; assigning tasks for processing said input data segments to worker nodes, where each worker node is assigned a task for processing an input data segment; determining, from data received from worker nodes executing said tasks, if a read pointer that points to a current read location in an input data segment processed by a task has not yet reached a predetermined threshold before input data segment end; and assigning of a new task to a free worker node, the new task being attributed a portion, referred to as split portion, of the input data segment that has not yet been processed by said task that has not yet reached a predetermined threshold before input data segment end, said split portion being a part of said input data segment that is located after said current read pointer location.