Defining a web crawl space

摘要

Provided are techniques for defining a web crawl space to be crawled. A seed list including one or more seed names is received from a user, wherein each seed name represents a website. In response to receiving the seed list, a web crawl space for the received seed list is generated by generating one or more allow rules.