Restricting network spidering

摘要

Disclosed are various embodiments for a request monitoring application. Content identifiers are generated according to a known sequence or pattern. Requests for invalid content identifiers are logged. Response behavior to subsequent requests is modified when a number of requests from a source having invalid content identifiers meets a threshold.

主权项

1. A non-transitory computer-readable medium embodying a program executable in at least one computing device, the program, when executed, causing the at least one computing device to at least:
randomly generate a plurality of content identifiers, each of the plurality of content identifiers comprising a respective key value identifying a respective one of a plurality of content entries, and the plurality of content entries being indexed according to the respective key value; obtain a request from a client corresponding to a network address; determine that the network address is included in a plurality of validated network addresses; determine that a quantity of a plurality of previously obtained requests from the network address and having a requested content identifier excluded from the plurality of content identifiers meets a threshold; perform, responsive to the quantity meeting the threshold and the network address being excluded from the valid network addresses, an action in response to the request; and wherein the action comprises at least one of denying the request, restricting a bandwidth to the network address, or communicating a response data comprising simulated content data to the network address.