Web crawling

摘要

The present invention is directed to mechanisms for improving the "crawling" of resources on a network, which takes into account the notion of browser state. An improved indexing scheme for the crawled results and improved search mechanisms are also disclosed.