METHOD FOR SEGMENTING NON-SEGMENTED TEXT USING SYNTACTIC PARSE

摘要

Embodiments of the present invention provide a method and apparatus for segmenting text by providing orthographic (308) and inflectional variations (306) to a syntactic parser (316). Under the present invention, possible segments are first identified in the sequence of characters. At least two of the identified segments overlap each other. For a least one of the segments, an alternative sequence of characters is identified. In some cases, this alternative sequence is formed through inflectional morphology (306), which identifies a different lexical form for a word identified by the segment. In some cases, the alternative sequence represents an orthographic variant (308) of a word identified by the segment. The identified segments and the alternative segments are then passed to a syntactic analyzer (316), which produces one or more syntatic parses. The segments found in the resulting parses represent the segmentation of the input sequence of characters.