发明名称 DATA PROCESSING METHOD AND APPARATUS
摘要 A data processing method and apparatus. The method comprises: if the number of completed map tasks within a pre-set time period is greater than or equal to a pre-set starting threshold, acquiring the overheads of unassigned partitions in outcomes output from the completed map tasks within the pre-set time period and before the pre-set time period (S201); according to the overhead of each of the unassigned partitions and starting states of reduce tasks, assigning corresponding reduce tasks to N partitions with the highest overheads (S202), where N is an integer greater than or equal to 1; and pulling data of the N partitions to the reduce tasks corresponding to the partitions for processing (S203), which can improve the resource utilization rate, and assigning reduce tasks according to the overheads of the partitions makes the assignment more rational and improves the overall performance of a MapReduce system.
申请公布号 WO2017031961(A1) 申请公布日期 2017.03.02
申请号 WO2016CN74042 申请日期 2016.02.18
申请人 HUAWEI TECHNOLOGIES CO., LTD. 发明人 LI, Yong;ZHU, Junhua
分类号 G06F9/50 主分类号 G06F9/50
代理机构 代理人
主权项
地址