Methods and apparatus for user-centered web crawling

摘要

Techniques are provided for user-centered search and crawling on an information network such as the world wide web. The techniques identify the nature of the web pages which are most relevant to a given predicate. The behavior of users is used to identify and determine the web pages which are most relevant to a specific crawl. Thus, the techniques are implemented in a web crawling system which can obtain the web pages specific to a given topic by leveraging the nature of the interests of the users in different topics.