DISCRIMINATIVE DATA SELECTION FOR LANGUAGE MODELING

摘要

A computer system for language modeling may collect training data from one or more information sources, generate a spoken corpus containing text of transcribed speech, and generate a typed corpus containing typed text. The computer system may derive feature vectors from the spoken corpus, analyze the typed corpus to determine feature vectors representing items of typed text, and generate an unspeakable corpus by filtering the typed corpus to remove each item of typed text represented by a feature vector that is within a similarity threshold of a feature vector derived from the spoken corpus. The computer system may derive feature vectors from the unspeakable corpus and train a classifier to perform discriminative data selection for language modeling based on the feature vectors derived from the spoken corpus and the feature vectors derived from the unspeakable corpus.

主权项

1. A computer system for language modeling, the computer system comprising:
a processor configured to execute computer-executable instructions; and memory storing computer-executable instructions configured to:
collect training data from one or more information sources;generate a spoken corpus containing text of transcribed speech;generate a typed corpus containing typed text;derive feature vectors from the spoken corpus;analyze the typed corpus to determine feature vectors representing items of typed text;generate an unspeakable corpus by filtering the typed corpus to remove each item of typed text represented by a feature vector that is within a similarity threshold of a feature vector derived from the spoken corpus;derive feature vectors from the unspeakable corpus; andtrain a classifier based on the feature vectors derived from the spoken corpus and the feature vectors derived from the unspeakable corpus.