Optimizing execution and resource usage in large scale computing

摘要

A method for tuning workflow settings in a distributed computing workflow comprising sequential interdependent jobs includes pairing a terminal stage of a first job and a leading stage of a second, sequential job to form an optimization pair, in which data segments output by the terminal stage of the first job comprises data input for the leading stage of the second job. The performance of the optimization pair is tuned by determining, with a computational processor, an estimated minimum execution time for the optimization pair and increasing the minimum execution time to generate an increased execution time. The method further includes calculating a minimum number of data segments that still permit execution of the optimization pair within the increased execution time.

主权项

1. A method for tuning workflow settings in a distributed computing workflow comprising sequential interdependent jobs, the method comprising:
pairing a terminal stage of a first job and a leading stage of a second, sequential job to form an optimization pair, in which data segments output by the terminal stage of the first job comprises data input for the leading stage of the second job; tuning a performance of the optimization pair by:
determining, with a computational processor, an estimated minimum execution time for the optimization pair;increasing the minimum execution time to generate an increased execution time; andcalculating, with the computational processor, a minimum number of data segments produced by the terminal stage that still permit execution of the optimization pair within the increased execution time; and executing, by distributed computing devices, the optimization pair to produce the minimum number of data segments.