| 摘要 |
A method, apparatus and article of manufacture for determining annotation quality, including obtaining N annotations on an artifact, wherein each annotation includes a feature and N annotations include three or more annotations including an annotation provided by a first human annotator, zero or more human annotations and zero or more imposter annotations, selectively displaying the N annotations to the first human annotator, wherein the annotation provided by the first human annotator is completely visible and each of the other N−1 annotations includes a feature hidden, and determining to annotation quality of one of the N−1 annotations based on input from the first human annotator regarding the displayed annotations, wherein ability of the first human annotator to identify imposter annotations or recognize that no imposter annotation exists is gated by the quality of the first human's annotation via hiding other annotations and requiring a probe based on the first human annotator's annotation.
|
