OPTIMIZATION OF TEXT-BASED TRAINING SET SELECTION FOR LANGUAGE PROCESSING MODULES

摘要

A device and a method provide for selection of a database from a corpus using an optimization function. The method includes defining a size of a database, calculating a distance using a distance function for each pair' in a set of pairs, and executing an optimization function using the distance to select each entry saved in the database until the number of saved entries equals the size of the database. Each pair in the set of pairs includes either two entries selected from a corpus or one entry selected from a set of previously selected entries and another entry selected from a set of a remaining portion of the corpus;. The distance function may be a Levenshtein distance function or a generalized Levenshtein distance function.