STAGE-AWARE PERFORMANCE MODELING FOR COMPUTER CLUSTER SIZING

摘要

A method, apparatus, and computer program product for configuring a computer cluster. Job information identifying a data processing job to be performed is received by a processor unit. The data processing job to be performed comprises a plurality of stages. Cluster information identifying a candidate computer cluster is also received by the processor unit. The processor unit identifies stage performance models for modeled stages that are similar to the plurality of stages. The processor unit predicts predicted stage performance times for performing the plurality of stages on the candidate computer cluster using the stage performance models and combines the predicted stage performance times for the plurality of stages to determine a predicted job performance time. The predicted job performance time may be used to configure the computer cluster.

主权项

1. A method of configuring a computer cluster, comprising:
receiving, by a processor unit, job information identifying a data processing job to be performed, wherein the data processing job to be performed comprises a plurality of stages; receiving, by the processor unit, cluster information identifying a candidate computer cluster; identifying, by the processor unit, stage performance models for modeled stages that are similar to the plurality of stages; predicting, by the processor unit, predicted stage performance times for performing the plurality of stages on the candidate computer cluster using the stage performance models; combining, by the processor unit, the predicted stage performance times to determine a predicted job performance time; and using the predicted job performance time to configure the computer cluster.