Credibility of text analysis engine performance evaluation by rating reference content

摘要

Evaluating the performance of a text analysis engine is provided. A plurality of pre-annotated reference documents and a set of annotation types associated with the pre-annotated reference documents are received. Annotation contexts of reference annotations in the plurality of pre-annotated reference documents are analyzed using the set of annotation types. Similar annotation contexts are identified between the reference annotations and the set of annotation types. Responsive to identifying the similar annotation contexts, the similar annotation contexts are clustered thereby forming a plurality of reference annotation clusters. A set of reference content heterogeneity scores are computed based on the number of reference annotation clusters for each annotation type in the set of annotation types. An integral reference content rate for the set of annotation types is then computed and output to a user.

主权项

1. A method, in a data processing system, for evaluating the performance of a text analysis engine, the method comprising:
receiving a plurality of pre-annotated reference documents; receiving a set of annotation types associated with the pre-annotated reference documents; analyzing annotation contexts of reference annotations in the plurality of pre-annotated reference documents using the set of annotation types; identifying similar annotation contexts between the reference annotations and the set of annotation types; responsive to identifying the similar annotation contexts, clustering the similar annotation contexts thereby forming a plurality of reference annotation clusters; computing a set of reference content heterogeneity scores based on the number of reference annotation clusters for each annotation type in the set of annotation types; computing an integral reference content rate for the set of annotation types; and outputting the integral reference content rate to a user.