Backfill scheduling for embarrassingly parallel jobs

摘要

Backfill scheduling for embarrassingly parallel jobs. A disclosed method includes: receiving an initial schedule having a plurality of jobs scheduled over time on a plurality of nodes, determining that a first job can be split into a plurality of sub-tasks that can respectively be performed in parallel on different nodes, splitting the first job into the plurality of sub-tasks, and moving a first sub-task from its position in the initial schedule to a new position to yield a first revised schedule.

主权项

1. A computer program product comprising software stored on a non-transitory computer storage medium, the software comprising:
first program instructions programmed to receive an initial schedule of jobs, including a plurality of jobs, scheduled over time on a plurality of nodes; second program instructions programmed to determine that a future job from the plurality of jobs can be split into a plurality of sub-tasks that can respectively be performed in parallel on different nodes, each sub-task of the plurality of sub-tasks being defined individually by corresponding pre-defined durations and corresponding sub-task node counts; third program instructions programmed to determine a backfill window in the initial schedule of jobs, the backfill window having a window duration and a window node count; fourth program instructions programmed to separate the future job into the plurality of sub-tasks by their corresponding pre-defined durations; fifth program instructions programmed to generate a set of backfill sub-tasks from the plurality of sub-tasks based on the corresponding pre-defined durations of each sub-task of the plurality of sub-tasks, each backfill sub-task having a combined pre-defined duration matching the window duration, and having sub-task node counts that match the window node count; sixth program instructions programmed to remove the future job from the initial schedule of jobs; seventh program instructions programmed to add the set of backfill sub-tasks into the backfill window to yield a revised schedule of jobs; and eighth program instructions programmed to execute the plurality of jobs and the set of backfill sub-tasks according to the revised schedule of jobs.