SYSTEM FOR WEB CRAWLING AND METHOD THEREOF

摘要

PURPOSE: A system for web crawling and a method thereof are provided to remarkably reduce the time taken for web crawling by simultaneously downloading external link pages linked to the webpage which has the highest access probability. CONSTITUTION: A seed page priority assigner(11) sets up standard seed pages for web crawling, produces access probability of the seed pages detected through the web crawling and gives priority to the seed page. A downloader(12) downloads the seed page having the highest priority and outlink pages linked to the seed page collectively. An outlink page priority assigner(13) produces access possibility of the seed page and gives the priority to an external link page.