Method and apparatus for processing text with variations in vocabulary usage

摘要

Text is processed to construct a model of the text. The text has a shared vocabulary. The text is partitioned into sets and subsets of texts. The usage of the shared vocabulary in two or more sets is different, and the topics of two or more subsets are different. A probabilistic model is defined for the text. The probabilistic model considers each word in the text to be a token having a position and a word value, and the usage of the shared vocabulary, topics, subtopics, and word values for each token in the text are represented using distributions of random variables in the probabilistic model, wherein the random variables are discrete. Parameters are estimated for the model corresponding to the vocabulary usages, the word values, the topics, and the subtopics associated with the words.

主权项

1. A computer-implemented method for processing text to construct a model of the text, comprising executing on a processor the steps of:
acquiring an electronic communication containing the text, wherein the text has a shared vocabulary, wherein the text includes words, wherein the text is partitioned into sets of texts and at least one set of text is partitioned into subsets of texts, wherein a usage of the shared vocabulary in two or more sets is different, and the topics of two or more subsets are different; defining a probabilistic model for the text, wherein the probabilistic model is stored in a memory operatively connected to the processor, and wherein the probabilistic model considers each word in the text to be a token having a position and a word value, and the usage of the shared vocabulary, topics, subtopics, and word values for each token in the text are represented using distributions of random variables in the probabilistic model, wherein the random variables are discrete, wherein each set of text has a vocabulary usage random variable, wherein each token is associated with the random variables corresponding to the topics, the subtopics, and the word values, wherein the distribution of the random variable associated with the topic for the token is dependent on the subset of text including the token, the distribution of the random variable associated with the subtopic for the token is dependent on the topic of the token, and the distribution of the random variable for the word value of the token is dependent on the associated subtopic and the vocabulary usage of the set of texts including the token; estimating parameters of the probabilistic model, based on the vocabulary usages, the word values, the topics, and the subtopics associated with the words; and classifying a text input using the probabilistic model, wherein the classifying includes one or combination of a dialect estimation, a topic estimation and a document retrieval, wherein for the dialect estimation, the text input is used in conjunction with the estimated parameters of the probabilistic model to compute dialect scores for estimating a dialect class, wherein for the topic estimation, the text input is used in conjunction with the estimated parameters of the probabilistic model to compute topic scores for estimating a topic class, and wherein for the document retrieval, the text input is used in conjunction with the estimated parameters of the probabilistic model to compute document scores for estimating a matching document.