ONLINE PRODUCT TESTING USING BUCKET TESTS

摘要

The technologies described herein use a statistical test to determine whether differences between data sets of buckets in a bucket test, such as differences between averages of two buckets (e.g., differences between means of two buckets), are directionally larger than a predetermined or preset minimum threshold value. The statistical test may also provide an extension to specify the minimum threshold value as a percentage. Also, described herein are techniques for estimating different control variables of a bucket test, such as estimating minimum bucket size to provide sufficient statistical power with use of the minimum threshold value.

主权项

1. Testing circuitry for bucket testing, comprising:
threshold metric circuitry configured to store a threshold metric of a bucket test of an update to an online product, wherein the threshold metric includes a software metric associated with the online product; minimum difference circuitry configured to store a predetermined minimum difference of the threshold metric; and confidence circuitry configured to store a confidence interval, a p-value, a test conclusion, or any combination thereof of the threshold metric.