Named entity extraction from a block of text

摘要

A data processing method, program, and apparatus for identifying a document within a block of text. A block of text is tokenized into a plurality of text tokens according to at least one rule parser. Each of the plurality of text tokens is sequentially compared to a plurality of document tokens to determine if the text token matches one of the plurality of document tokens. The plurality of document tokens correspond to a plurality of documents which have been tokenized according to the one or more rule parsers. Each matched text token is filtered according to predetermined filtering criteria to generate one or more candidate text tokens. It is then determined whether sequence of candidate text tokens that occur in sequential order within the block of text match sequence of document tokens. If so, then it is determined that the document has been identified within the block of text. The document can correspond to an artist, a song names, and misspellings and aliases thereof.

主权项

1. A data processing method of identifying a document within a block of text, comprising:
tokenizing a block of text into a plurality of text tokens according to at least one rule parser; comparing each of the plurality of text tokens to a plurality of document tokens sequentially and determining if the text token matches one of the plurality of document tokens, wherein the plurality of document tokens correspond to a plurality of documents which have been tokenized according to the at least one rule parser; filtering each matched text token according to predetermined filtering criteria to generate one or more candidate text tokens; and matching a sequence of candidate text tokens from the one or more candidate text tokens to a sequence of document tokens, wherein the sequence of candidate text tokens occurs in sequential order within the block of text, wherein the predetermined filtering criteria includes a plurality of stop words and a plurality of common words, and wherein the filtering includes performing a stop word analysis of comparing the matched text token to the plurality of stop words, such that (i) if the matched text token matches one of the plurality of stop words, a next text token in sequential order, of the plurality of text tokens, is compared to the plurality of document tokens, and (ii) if the matched text token does not match one of the plurality of stop words, a common word analysis of comparing the matched text token to the plurality of common words is performed, such that
(a) if the matched text token matches one of the plurality of common words and is not capitalized, the next text token in sequential order is compared to the plurality of document tokens, and(b) if the matched text token does not match one of the plurality of common words, the matched text token is identified as a candidate text token, and the next text token in sequential order is compared to the plurality of document tokens.