HYBRID PRE-TRAINING OF DEEP BELIEF NETWORKS

摘要

Pretraining for a DBN initializes weights of the DBN (Deep Belief Network) using a hybrid pre-training methodology. Hybrid pre-training employs generative component that allows the hybrid PT method to have better performance in WER (Word Error Rate) compared to the discriminative PT method. Hybrid pre-training learns weights which are more closely linked to the final objective function, allowing for a much larger batch size compared to generative PT, which allows for improvements in speed; and a larger batch size allows for parallelization of the gradient computation, speeding up training further.

主权项

1. In a deep belief network (DBN), a method for training the DBN for operational decoding comprising:
pre-training initial weights based on determined optimal weights; identifying, during pre-training, a batch size and number of parameters corresponding to the initial weights, the batch size larger than that allowable for purely generative pretraining; and invoking a plurality of machines for performing training in parallel fashion.