| 主权项 |
1. An apparatus for searching at least one web site, comprising:
a processor coupled to a display device, the processor containing instructions for a web crawler, wherein the web crawler comprises a parallel web crawler comprising the following operations: 1) searching all pages on the at least one web site; 2) searching at least one search engine index for said all pages on the at least one web site; 3) determining if said all pages on the at least one web site are cached on the at least one search engine index; 4) repeating the searching of the at least one search engine index after a specified period of time, and determining what changes have occurred; 5) determining, from the at least one web site, all anchor text links, internal links, and external links, and displaying the anchor text links, internal links, and external links in a report, wherein the anchor text links are used to predict the similarity of a page to a query before the page is downloaded, wherein the predicting of the similarity is based on the anchor text links and the predicting allows the processor to engage in focused crawling; 6) searching the at least one web site for broken links; 7) searching the at least one web site for leading links; 8) searching the at least one web site for external links; 9) searching the at least one web site and displaying all image links; 10) extracting a specified type of data from the at least one web site; 11) copying and indexing source code from the at least one web site; 12) blocking directories, pages, and sections from the at least one web site during a search; and 13) selecting only static pages to search on the at least one web site, wherein the web crawler is executed with these thirteen operations being selected; wherein each of these thirteen operations, starting at the searching all pages on the at least one web site, are run in parallel, wherein the web crawler results in up to date data for said all pages, the web crawler results in gathering information from said all pages, and the web crawler results in automated browsing and maintaining of links and HTML code for the at least one website, the processor further comprising a politeness policy that provides guidelines for avoiding overloading web pages of the at least one web site revisited by the web crawler, wherein the politeness policy is based on Universal Resource Locator (URL) normalization or URL canonicalization of at least one URL of the at least one web site; the processor further comprising a re-visit policy that dictates when to check for changes to pages of the at least one web site already examined, wherein the revisit policy comprises a combination of uniform and proportional policies that monotonically and sub-linearly increase with rate of change, wherein the revisit policy comprises a binary measure that indicates whether a page of the at least one web site is accurate, wherein the revisit policy comprises a measure of how outdated the page is, wherein the revisit policy maintains a high value of average freshness by ignoring pages of the at least one web site that change to often; the processor further comprising a selection policy searching all pages on the at least one web site, wherein the using of the anchor text links to predict the similarity of the page is based on the selection policy; the processor further comprising a parallelization policy for these thirteen operations, wherein coordination of these thirteen operations is based on the parallelization policy. |
