Identifying high impact bugs

摘要

Test cases are executed by the software engineering test system. The test cases target software products. Test outputs are generated indicating whether the software engineering test system determined the test cases to have passed or failed. Separately, bug records are stored in a first dataset whose records identify corresponding bugs. Records of the test case executions are stored in a second dataset. Records thereof indicate whether a corresponding test case failed when executed. Such records may have bug identifiers entered by a test engineer and corresponding to bugs identified by the test engineer. The first dataset is parsed to identify records of test runs that have failed, and for each such test run record a bug identifier thereof is identified. Statistics such as failure counts are updated for the bugs found in the test run records.

主权项

1. A method of identifying high impact bugs detected by a software engineering test system comprised of one or more computing devices, the method, performed by the one or more computing devices, comprising:
executing runs of the test cases by the software engineering test system, the test cases authored by test engineers, the test cases targeting software products to be tested, wherein, when test case runs are executed by the software engineering test system, corresponding software products are executed by the software engineering test system and corresponding test outputs are generated indicating whether the software engineering test system determined the test cases to have passed or failed; storing bug records in a first data store, each bug record comprising an identifier of a corresponding bug and metadata about the bug entered by a test engineer, wherein at least some of the bug records correspond to failures of the software products that did not occur while the software products were being tested; storing, in a second data store, test run records of the outputs of the runs of the tests cases, the test run records comprising records of the test reports, respectively, each test run record indicating whether a corresponding single test case execution failed, each test run record corresponding to only one execution of a corresponding single test case, each test run record comprising one or more of the bug identifiers entered by a test engineer to indicate that a corresponding one of the bug records was identified by the test engineer as a cause of failure of a corresponding test run, wherein at least some of the test run records comprise respective pluralities bug identifiers such that some of the test run records are associated with multiple distinct bugs; computing failure statistics for the bug identifiers, respectively, by crawling the first data store to search for test run records that indicate that corresponding test runs failed, determining numbers of occurrences of the respective bug identifiers within the results, and computing the failure statistics according to the respective numbers of occurrences of the respective bug identifiers, wherein each failure statistic indicates the determined number of occurrences of a respective one of the bug identifiers; and determining which of the bugs are systemic to the software engineering test system by categorizing the failure statistics according to the software products that the corresponding bugs are associated with.