LONG TERM METRICS APPLIED TO MULTIVARIATE TESTING

摘要

Embodiments provide techniques for testing a plurality of variations of a user experience on a subscription-based online site, where each of the plurality of variations is distinct from other variations in the plurality of variations. Each of a plurality of users is assigned to one of the plurality of variations. For each of a plurality of user requests received during a predetermined window of time, embodiments determine one of the plurality of variations for use in processing the user request, based on a user associated with the user request, and process the respective user request using the determined variation. Long-term user metric information relating to subscription services for the subscription-based online site is determined, with respect to the plurality of users. Embodiments then rate at least one of the plurality of variations, based on the determined long-term user metric information.

主权项

1. A method of testing a plurality of variations of a user experience on a subscription-based online site, wherein each of the plurality of variations is distinct from other variations in the plurality of variations, comprising:
assigning each of a plurality of users to one of the plurality of variations of the user experience; for each of a plurality of user requests received from the plurality of users during a predetermined window of time:
determining one of the plurality of variations for use in processing the user request, based on a user associated with the user request; andprocessing the respective user request using the determined variation; and determining long-term user metric information relating to subscription services for the subscription-based online site, with respect to the plurality of users, and for each of the plurality of variations.