Adaptive parallel data processing

摘要

Described herein are methods, systems, apparatuses and products for adaptive parallel data processing. An aspect provides providing a map phase in which at least one map function is applied in parallel on different partitions of input data at different mappers in a parallel data processing system; providing a communication channel between mappers using a distributed meta-data store, wherein said map phase comprises mapper data processing adapted responsive to communication with said distributed meta-data store; and providing data accessible by at least one reduce phase node in which at least one reduce function is applied. Other embodiments are disclosed.

主权项

1. A computer program product comprising:
a non-transitory computer readable storage medium having computer readable program code embodied therewith, the computer readable program code comprising: computer readable program code configured to provide a map phase in which at least one map function is applied in parallel on different partitions of input data at different mappers in a parallel data processing system, and to perform the map phase; computer readable program code configured to provide a communication channel which enables communication between mappers using a distributed meta-data store; wherein to perform the map phase comprises:
using the communication channel to permit each mapper to:
post meta-data for use by at least one other mapper about its state; andreceive information regarding a state of at least one other mapper; andperforming mapper data processing which is adapted responsive in response to the communication which takes place between mappers via the distributed meta-data store; and computer readable program code configured to provide data accessible by at least one reduce phase node in which at least one reduce function is applied.