GENERATING GOLD QUESTIONS FOR CROWDSOURCING

摘要

A system and method for generating gold questions for labeling tasks are disclosed. The method includes sampling a positive class from a predefined set of classes to be used in labeling documents, based on a computed measure of class popularity. A set of negative classes is identified from the set of classes based on a distance measure between the positive class and other classes in the set of classes. A gold question is generated which includes a document representative of the positive class and a set of candidate answers. The candidate answers include a label for the positive class and a label for each of the negative classes in the identified set of negative classes. A task may be generated which includes the gold question and a plurality of standard questions which each include a document to be labeled. A computer processor may implement all or part of the method.

主权项

1. A method for generating a gold question for a labeling task comprising:
sampling a positive class from a predefined set of classes to be used in labeling documents, based on a computed measure of class popularity; for the positive class, identifying a set of negative classes from the set of classes based on a distance measure between the positive class and other classes in the set of classes; generating a gold question which includes a document representative of the positive class and a set of candidate answers, the candidate answers including a label for the positive class and a label for each of the negative classes in the identified set of negative classes; and outputting the gold question, wherein at least one of the sampling, identifying, and generating is performed with a computer processor.