Determining similarity of linguistic objects

摘要

A computer-implemented system for searching includes a data store accessible via a network for storing a data set; an indexing system coupled to the network and indexing the data set, the indexing system configured to generate content vectors for terms in the data set; generate index vectors for terms in the data set; and generate a bitset signature from the index vector. The system further includes a search module coupled to the network and configured to receive a search query and perform a search on one or more terms in the search query by accessing a bitset signature and content vector corresponding to the term; retrieving bitset signatures that are within a predetermined closeness to the bitset signature; selecting content vectors corresponding to retrieved bitset signatures; and selecting content vectors that are within a predetermined similarity to the term content vector; and return the terms corresponding to the content vectors.

主权项

1. A computer-implemented system for searching, comprising:
a data store accessible via a network for storing a data set; an indexing system coupled to the network and indexing the data set, the indexing system including a processor configured to:
generate content vectors for terms in the data set, wherein the content vectors define a similarity metric;generate index vectors for the terms in the data set from the content vectors to access the terms in the data set; andgenerate bitset signatures from the index vectors to determine similarity with the terms in the data set, wherein the bitset signatures include a first section for positive magnitude values and a second section for negative magnitude values, and generating the bitset signatures from the index vectors comprises:
for each of a predetermined number of highest magnitude positive values in the corresponding index vector, setting a corresponding bitset signature value at a corresponding dimension in the first section to a predetermined value; andfor each of a predetermined number of highest magnitude negative values in the corresponding index vector, setting a corresponding bitset signature value at a corresponding dimension in the second section to the predetermined value; and a search module coupled to the network and including a processor configured to receive a search query and perform a search on each of one or more terms in the search query by:
accessing a bitset signature and content vector corresponding to a term in the search query;retrieving bitset signatures that are within a predetermined closeness to the accessed bitset signature;selecting content vectors corresponding to the retrieved bitset signatures;identifying the selected content vectors that are within a predetermined similarity to the accessed content vector corresponding to the term in the search query; andreturning the terms of the data set corresponding to the identified content vectors.