| 摘要 |
Described is a technology by which an assignment model is computed to distribute labeling tasks among judging entities (judges). The assignment model is optimized by obtaining accuracy-related data of the judges, e.g., by probing the judges with labeling tasks having a gold standard label and evaluating the judges' labels against the gold standard labels, and optimizing for accuracy. Optimization may be based upon on or more other constraints, such as per-judge cost and/or quota. |
