Timing module for regulating hits by a spidering engine

摘要

A method and system for retrieving web site based information by a spider engine at a target bandwidth is described. A target bandwidth is received from the spider engine. A wait time is calculated by a timing module. Data retrieval from a web site is delayed by the calculated wait time so that data is retrieved at the desired target bandwidth.