CROWD WORKER QUALITY IMPROVEMENT WITH CONTINUING TRAINING

摘要

Workers are trained and assisted to improve quality of task results generated by them. Initially, workers seeking to perform intelligence tasks within the context of a crowdsourcing system are provided with offline training. Subsequently, workers are qualified, with different nuances of the performance of intelligence tasks being assessed independently. A task owner specifies minimum quality thresholds, including separately specified thresholds for different nuances. Qualified workers are assigned to perform intelligence tasks, thereby producing useful work. While doing so, their quality is monitored, and online training is provided to aid workers with specific nuances of the intelligence tasks where the workers' performance is suboptimal. Workers whose quality drops below minimum quality thresholds are aided in improving their quality, including being prevented from performing intelligence tasks for a predefined period of time, being mandated to perform offline training, or combinations thereof.

主权项

1. A computing device for improving quality of results to intelligence tasks, the results being generated by a human worker within a crowdsourcing system, the computing device comprising one or more processing units and computer-readable media comprising computer-executable instructions that, when executed by the processing units, cause the computing device to perform steps comprising:
detecting that the quality of the results generated by the human worker are below a predetermined quality threshold; preventing, for a predetermined and finite period of time, the human worker from generating further results to intelligence tasks of a same overall task as the intelligence tasks to which the human worker generated the results that were detected to be below the predetermined quality threshold, the preventing being triggered by the detecting of the quality of the results being below the predetermined quality threshold; enabling the human worker to again perform useful work by receiving intelligence tasks and providing results thereto only in response to an expiration of the predetermined and finite period of time; detecting that the results generated by the human worker exceed the predetermined quality threshold but that the results generated by the human worker directed to at least one specific nuance of the intelligence tasks are of lesser quality than the results generated by the human worker directed other specific nuances of the intelligence tasks; and providing online training, while the human worker remains enabled to perform useful work, the online training being directed to the at least one specific nuance of the intelligence tasks, the providing being triggered by the detecting of results directed to the at least one specific nuance being of lesser quality than the results directed to the other specific nuances.