STREAMLINED CREATION AND UTILIZATION OF REFERENCE HUMAN INTELLIGENCE TASKS

摘要

Reference intelligence tasks are automatically generated for subsequent utilization in crowdsourced processing of intelligence tasks. Reference intelligence tasks are categorized into predetermined categories, including categories defined by an intended utilization of such intelligence tasks. A trusted set of workers are provided with intelligence tasks and, if a specified number of those trusted workers reach consensus as to what is an appropriate answer, then such an answer is a definitive answer. Conversely, if no consensus is initially reached, then additional trusted workers can be utilized to determine if, in combination, consensus can be reached. To frame categorization considerations by the trusted set of workers, they are provided with statistics relevant to categorizations. An automatic category assignment, or reassignment, can be performed if necessary. Additionally, a maximum amount of time by which the automatic generation of a set of reference intelligence tasks is to complete is, optionally, established.

主权项

1. One or more computer-readable media comprising computer-executable instructions for generating reference human intelligence tasks, the computer-executable instructions directed to steps comprising:
obtaining a set of human intelligence tasks; selecting a first human intelligence task from among the set of human intelligence tasks; presenting the first human intelligence task to a predetermined first quantity of workers; generating the first human intelligence task as a reference human intelligence task if greater than a first threshold quantity of workers, from among the predetermined first quantity of workers, identified a same answer to the first human intelligence task; presenting the first human intelligence task to additional workers, not exceeding a predetermined second quantity of workers, if a maximum number of workers of the predetermined first quantity of workers that identified the same answer to the first human intelligence task was less than the first threshold quantity of workers; and generating the first human intelligence task as a reference human intelligence task if greater than a second threshold quantity of workers, from among the predetermined first quantity of workers combined with the additional workers, identified the same answer to the first human intelligence task.