VALIDATING ADVERTISEMENT PREDICTIONS USING ADVERTISEMENT EXPERIMENTS

摘要

Experiment performance data associated with a first advertisement campaign is generated during a first advertisement campaign experiment. The experiment performance data includes a first set of experiment performance data and a second set of experiment performance data. A first set of simulated performance data for the first advertisement campaign using first set of auction instances with the advertisement campaign setting having second advertisement campaign value is generated. A second set of simulated performance data for the first advertisement campaign using the second set of auctions instances with the advertisement campaign setting having the first advertisement campaign value is generated. Accuracy of the first set of performance data and the second set of performance data is determined by comparing the first performance data to the second set of experiment performance data, and by comparing the second performance data to the first set of experiment performance data.

主权项

1. A computerized method of evaluating a simulation engine, the method comprising:
receiving, by a first computer from an experiment engine of a second computer via a network, experiment performance data associated with an advertisement campaign, the advertisement campaign including:
a first set of experiment performance data including a first number of selections generated during an actual advertisement campaign experiment from a first set of auction instances using the advertisement campaign, the advertisement campaign having a first advertisement campaign setting value; anda second set of experiment performance data including a second number of selections generated during the actual advertisement campaign experiment from a second set of auction instances using the advertisement campaign, the advertisement campaign having a second advertisement campaign setting value; generating, by a simulation engine of the first computer using the first set of auction instances after completion of the actual advertisement campaign experiment, a first set of simulated performance data for the advertisement campaign that includes a third number of selections simulated at the second advertisement campaign setting value; generating, by the simulation engine using the second set of auction instances after completion of the actual advertisement campaign experiment, a second set of simulated performance data for the advertisement campaign that includes a fourth number of selections simulated at the first advertisement campaign setting value; comparing, by one or more processors of the first computer, the third number of selections of the first set of simulated performance data at the second advertisement campaign setting value with the second number of selections of the second set of experiment performance data to generate a first metric; comparing, by one or more processors of the first computer, the fourth number of selections of the second set of simulated performance data at the first advertisement campaign setting value with the first number of selections of the first set of experiment performance data to generate a second metric; and determining, by one or more processors of the first computer, accuracy of the simulation engine based on the first comparison metric and the second comparison metric.