SYSTEM AND METHOD FOR LEARNING WORD EMBEDDINGS USING NEURAL LANGUAGE MODELS

摘要

A system and method are provided for learning natural language word associations using a neural network architecture. A word dictionary comprises words identified from training data consisting a plurality of sequences of associated words. A neural language model is trained using data samples selected from the training data defining positive examples of word associations, and a statistically small number of negative samples defining negative examples of word associations that are generated from each selected data sample. A system and method of predicting a word association is also provided, using a word association matrix including data defining representations of words in a word dictionary derived from a trained neural language model, whereby a word association query is resolved without applying a word position-dependent weighting.

主权项

1. A method of learning natural language word associations using a neural network architecture, comprising processor implemented steps of:
storing data defining a word dictionary comprising words identified from training data consisting a plurality of sequences of associated words; selecting a predefined number of data samples from the training data, the selected data samples defining positive examples of word associations; generating a predefined number of negative samples for each selected data sample, the negative samples defining negative examples of word associations, wherein the number of negative samples generated for each data sample is a statistically small proportion of the number of words in the word dictionary; and training a neural language model using said data samples and said generated negative samples.