BACKFILL SCHEDULING FOR EMBARRASSINGLY PARALLEL JOBS

摘要

Backfill scheduling for embarrassingly parallel jobs. A disclosed method includes: receiving an initial schedule having a plurality of jobs scheduled over time on a plurality of nodes, determining that a first job can be split into a plurality of sub-tasks that can respectively be performed in parallel on different nodes, splitting the first job into the plurality of sub-tasks, and moving a first sub-task from its position in the initial schedule to a new position to yield a first revised schedule.

主权项

1. A method comprising:
determining that a future job of an initial schedule of jobs can be split into a plurality of sub-tasks, each sub-task of the plurality of sub-tasks being suitable for parallel performance with other sub-tasks of the plurality of sub-tasks, each sub-task being defined individually by a corresponding pre-defined duration and having a sub-task node count; determining a backfill window in the initial schedule of jobs, the backfill window having a window duration and a window node count; separating the future job into the plurality of sub-tasks according to their corresponding pre-defined durations; generating a set of backfill sub-tasks from the plurality of sub-tasks, each backfill sub-task of the set of backfill sub-tasks having a combined pre-defined duration and a combined sub-task node count, the combined pre-defined duration matching the window duration and the combined sub-task node count that matches the window node count; removing the future job from the initial schedule of jobs to create a set of remaining jobs; adding the set of backfill sub-tasks into the backfill window of the initial schedule of jobs to create a revised schedule of jobs; and executing the set of remaining jobs and the set of backfill sub-tasks according to the revised schedule of jobs.