Performance regression testing in presence of noise

摘要

A method and system includes calculating a performance metric for each of a plurality of builds of a software application in view of a respective performance test result associated with each of the plurality of builds, calculating a respective difference in performance metrics for each pair of consecutive builds of the plurality of builds, determining a largest performance drop in view of respective differences in the performance metrics among the pairs of consecutive builds of the plurality of builds, wherein the largest performance drop is associated with a first pair of consecutive builds comprising a first build and a second build, determining, by a processing device, a confidence level associated with the largest performance drop in view of performance test results associated with the first build and the second build, in response to determining that the confidence level is greater than or equal to a threshold, identifying one of the first build or the second build as a problematic build of the software application.

主权项

1. A method comprising:
calculating a performance metric for each of a plurality of builds of a software application in view of a respective performance test result associated with each of the plurality of builds; calculating a respective difference in performance metrics for each pair of consecutive builds of the plurality of builds; determining a largest performance drop in view of respective differences in the performance metrics among the pairs of consecutive builds of the plurality of builds, wherein the largest performance drop is associated with a first pair of consecutive builds comprising a first build and a second build; determining, by a processing device, a confidence level associated with the largest performance drop in view of performance test results associated with the first build and the second build; and in response to determining that the confidence level is greater than or equal to a threshold, identifying one of the first build or the second build as a problematic build of the software application.