JOINT OPTIMIZATION OF MULTIPLE PHASES IN LARGE DATA PROCESSING

摘要

Methods and arrangements for task scheduling. A plurality of jobs is received, each job comprising at least a map phase, a copy/shuffle phase and a reduce phase. For each job, there are determined a map phase execution time and a copy/shuffle phase execution time. Each job is classified into at least one group based on at least one of: the determined map phase execution time and the determined copy/shuffle phase execution time. The plurality of jobs are executed via processor sharing, and the executing includes determining a similarity measure between jobs based on current job execution progress. Other variants and embodiments are broadly contemplated herein.

主权项

1. A method comprising:
utilizing at least one processor to execute computer code configured to perform the steps of: receiving a plurality of jobs, each job comprising at least a map phase, a copy/shuffle phase and a reduce phase; determining, for each job, a map phase execution time and a copy/shuffle phase execution time; classifying each job into at least one group based on at least one of: the determined map phase execution time and the determined copy/shuffle phase execution time; and executing the plurality of jobs via processor sharing; said executing comprising determining a similarity measure between jobs based on current job execution progress.