IDENTIFYING UNVISITED PORTIONS OF VISITED INFORMATION

摘要

An illustrative embodiment for identifying unvisited portions of visited information to visit, receives information to crawl, wherein the information is representative of one of web based information and non-web based information, computes a locality sensitive hash (LSH) value for the received information and identifies a most similar information visited thus far. The illustrative embodiment determines whether the LSH of the received information is equivalent to most similar information visited thus far and responsive to a determination that the LSH of the received information is not equivalent to most similar information visited thus far, identifies a visited portion of the received information using information for most similar information visited thus far and crawls only unvisited portions of the received information.