LEVERAGING ANNOTATION BIAS TO IMPROVE ANNOTATIONS

摘要

In order to leverage annotation bias in batch annotations, obtained via crowdsourcing, on a set of comments on user posts in a social network, a system may select a subset of the comments for annotation based on how informative expected annotations for the comments in the subset are for the one or more classifiers and probabilities of occurrence of the expected annotations based on a predetermined annotation probability distribution. Note that the classifier may predict how likely the expected annotations are accurate for the comments in a given subset. Moreover, the predetermined annotation probability distribution may specify the annotation bias. In this way, the system may use the annotation bias to select the subset that is likely to receive expected annotations and, thus, are that are easier to use in training the classifier.

主权项

1. A computer-implemented method for selecting a subset of a set of comments associated with a group of documents, the method comprising:
accessing, at memory locations, the set of comments and a predetermined annotation probability distribution of annotations for another set of comments associated with another group of documents, wherein the annotation probability distribution specifies biases in the annotations for the other set of comments; and using a computer processor that is coupled to the memory location and programmed to select the subset:
selecting the subset based on how informative expected annotations for the comments in the subset are for a classifier and probabilities of occurrence of the expected annotations based on the predetermined annotation probability distribution, wherein the classifier predicts how likely the expected annotations are accurate for the comments in the subset.