Compiler-guided software accelerator for iterative HADOOP® jobs

摘要

Various methods are provided directed to a compiler-guided software accelerator for iterative HADOOP® jobs. A method includes identifying intermediate data, generated by an iterative HADOOP® application, below a predetermined threshold size and used less than a predetermined threshold time period. The intermediate data is stored in a memory device. The method further includes minimizing input, output, and synchronization overhead for the intermediate data by selectively using at any given time any one of a Message Passing Interface and Distributed File System as a communication layer. The Message Passing Interface is co-located with the HADOOP® Distributed File System.

主权项

1. A method, comprising:
identifying a set of map tasks and reduce tasks capable of being reused across multiple iterations of an iterative HADOOP® application; and reducing a system load imparted on a computer system executing the iterative HADOOP® application by transforming a source code of the iterative HADOOP® application to launch the map tasks in the set only once and keep the map tasks in the set alive for an entirety of the execution of the iterative HADOOP® application; wherein the map tasks in the set are kept alive for the entirety of the execution by guarding an invocation to a runjob( ) function beginning at a first iteration of the iterative HADOOP® application to prevent a re-launching of any of the maps tasks and reduce tasks in the set in subsequent iterations of the iterative HADOOP® application, the invocation to the runJob( ) function is guarded by a flag, which is set to true for the first iteration and false for the subsequent iterations.