APPARATUS AND METHOD FOR DECODING USING JOINT TOKENIZATION AND TRANSLATION

摘要

PURPOSE: An integration decoding device and a method thereof for integrating a token separation process and a translation process are provided to reduce translation error and generate candidate tokens. CONSTITUTION: A candidate token generation unit(210) generates a candidate token by applying a maximum entropy model to an input character sequence. A probability calculation unit(220) calculates token separation probability of the candidate tokens through a language model. An OOV(Out Of Vocabulary) processing unit(230) processes OOV by using word frequency and discount information. A translation unit(240) generates a translated word according to the translation model through an OOV processing result.