BUDGET OPTIMAL CROWDSOURCING

摘要

To optimize the number of correct decisions made by a crowdsourcing system given a fixed budget, tasks for multiple decisions are allocated to workers in a sequence. A task is allocated to a worker based on results already achieved for that task from other workers. Such allocation addresses the different levels of difficulty of decisions. A task also can be allocated to a worker based on results already received for other tasks from that worker. Such allocation addresses the different levels of reliability of workers. The process of allocating tasks to workers can be modeled as a Bayesian Markov decision process. Given the information already received for each item and worker, an estimate of the number of correct labels received can be determined. At each step, the system attempts to maximize the estimated number of correct labels it expects to have given the inputs so far.

主权项

1. A computer-implemented process, comprising:
accessing data describing a plurality of decisions, each decision having an associated task, each task having an associated cost; accessing data describing a plurality of individuals; selecting a task for one of the plurality of decisions and one of the plurality of individuals based on results already achieved for the tasks as already performed by other of the plurality of individuals, by maximizing an estimated number of correct decisions given a budget; delivering a request to perform the task for the selected decision to a computer associated with the selected individual; receiving a result for the task from the computer associated with the selected individual; and repeating the steps of selecting, delivering and receiving until the budget is exhausted.