TIMING MODULE FOR REGULATING HITS BY A SPIDERING ENGINE

摘要

A method and system for retrieving web-site based information by a spider engine at a target bandwidth is described. A target bandwidth is received from the spider engine. A wait time is calculated by a timing module. Data retrieval from a web site is delayed by the calculated wait time so that data is retrieved at the desired target bandwidth.