Regression evaluation using behavior models of software applications

摘要

Comparisons of different versions of an application may be compared using a behavior model of the application. A behavior model may be derived from n-gram analysis of observations of the application in production. The behavior model may include sequences of inputs received by the application or functions performed by the application, where each sequence is an n-gram observed in tracer data. Each n-gram may be coupled with a resource consumption to give a behavior model with performance data. A regression analysis may apply a behavior model derived from a first version of an application to the performance observations of a new version to create an expected performance metric for the new version. A similarly calculated metric from a previous version may be compared to the metric from a new version to determine an improvement or degradation of performance.

主权项

1. A method performed by at least one hardware processor, said method comprising:
receiving a first behavior model representing a frequency of observation of each of a first plurality of n-grams during execution of a first version of an application in a production environment, each n-gram in the first plurality of n-grams identifying a different sequence of one or more of (i) two or more function sequences observed during execution of the first version of the application in the production environment, or (ii) two or more input sequences input to the first version of the application during execution of the first version of the application in the production environment; generating a first performance metric for the first version of the application based on the first behavior model, including (i) determining a first representative sample of n-grams from the first plurality of n-grams, (ii) determining a first corresponding performance metric for each n-gram in the first representative sample of n-grams, and (iii) summing each first corresponding n-gram performance metric for all of the n-grams in the first representative sample of n-grams to obtain the first performance metric; receiving a second behavior model representing a frequency of observation of each of a second plurality of n-grams during execution of a second version of the application in a testing environment, each n-gram in the second plurality of n-grams identifying a different sequence of one or more of (i) two or more function sequences observed during execution of the second version of the application in the testing environment, or (ii) two or more input sequences input to the second version of the application during execution of the second version of the application in the testing environment; generating a second performance metric for the second version of the application based on the second behavior model, including (i) determining a second representative sample of n-grams from the second plurality of n-grams, (ii) determining a second corresponding performance metric for each n-gram in the second representative sample of n-grams, and (iii) summing each second corresponding n-gram performance metric for all of the n-grams in the second representative sample of n-grams to obtain the second performance metric; and generating a visualization comparing performance of the first version of the application in the production environment with the second version of the application in the testing environment, the visualization including:
one or more histograms visualizing the first plurality of n-grams and the second plurality of n-grams; andone or more charts or graphs that present performance metrics for successive versions of the application, the one or more charts or graphs presenting at least the first performance metric and the second performance metric.