NOISE REMOVAL IN MULTIBYTE TEXT ENCODINGS USING STATISTICAL MODELS

摘要

Disclosed is a method of validating a byte sequence having a plurality of states, the method comprising designating one or more noise states from among the plurality of states; generating a most probable state sequence for the byte sequence; utilizing said state sequence to identify all noise in the byte sequence; and localizing said noise in said noise states. Once localized, the noise may be deleted from the byte sequence.