| 摘要 |
This invention provides a method and apparatus for automated search and retrieval processing that includes a tokenizer, a noun phrase analyzer, and a morphological analyzer. The tokenizer includes a parser that extracts characters from the stream of text, and identifying element for identifying a token formed of characters in the stream of text that include lexical matter, and a filter for assigning tags to those tokens requiring further linguistic analysis. The tokenizer, in a single pass through the stream of text, determines the further linguistic processing suitable to each particular token contained in the stream of text. The noun phrase analyzer annotates tokens with tags identifying characteristics of the tokens and contextually analyzes each token. During processing, the noun phrase analyzer can also disambiguate individual token characteristics and identify agreement between tokens. Themorphological analyzer organizes, utilizes, analyzes, and generates morphological data related to the tokens. In particular, the morphological analyzer locates a stored lexical expression representative of a candidate token found in a stream of natural language text, identifies a paradigm for the candidate token based upon the stored lexical expression, and applies transforms contained within the identified paradigm to the candidate token.
|
