Scalable lookup-driven entity extraction from indexed document collections

摘要

A set of documents is filtered for entity extraction. A list of entity strings is received. A set of token sets that covers the entity strings in the list is determined. An inverted index generated on a first set of documents is queried using the set of token sets to determine a set of document identifiers for a subset of the documents in the first set. A second set of documents identified by the set of document identifiers is retrieved from the first set of documents. The second set of documents is filtered to include one or more documents of the second set that each includes a match with at least one entity string of the list of entity strings. Entity recognition may be performed on the filtered second set of documents.

主权项

1. A method for filtering a set of documents, comprising:
receiving a list of entity strings; determining a set of token sets that covers the entity strings in the list, the number of tokens in the set of token sets being less than the number of words of the entity strings in the list of entity strings; querying an inverted index generated on a first set of documents using the set of token sets to determine a set of document identifiers for a subset of the documents in the first set; retrieving from the first set of documents a second set of documents, which is a subset of the first set of documents, identified by the set of document identifiers; and filtering the second set of documents to include one or more documents of the second set that each include a match with at least one entity string of the list of entity strings.