Distributed processing apparatus and method for processing large data through hardware acceleration

摘要

A distributed data processing apparatus and method through hardware acceleration are provided. The data processing apparatus includes a mapping node including mapping units configured to process input data in parallel to generate and output mapping results. The data processing apparatus further includes a shuffle node including shuffle units and a memory buffer, the shuffle units configured to process the mapping results output from the mapping units in parallel to generate and output shuffle results, and the shuffle node configured to write the shuffle results output from the shuffle units in the memory buffer. The data processing apparatus further includes a merge node including merge units configured to merge the shuffle results written in the memory buffer to generate merging results.

主权项

1. A data processing apparatus comprising:
a mapping node comprising mapping units configured to process input data in parallel to generate and output mapping results; a shuffle node comprising shuffle units and a memory buffer, the shuffle units configured to process the mapping results output from the mapping units in parallel to generate and output shuffle results, and the shuffle node configured to write the shuffle results output from the shuffle units in the memory buffer; a merge node comprising merge units configured to merge the shuffle results written in the memory buffer to generate merging results; and an input distribution node configured to distribute the input data among the mapping units on a record-by-record basis, wherein a number of the mapping units is determined based on a unit time taken by the input distribution node to input a record of the input data into one of the mapping units, or a unit time taken by the one of the mapping units to process the record, or any combination thereof.