PARTITIONING A SEARCH SPACE FOR DISTRIBUTED CRAWLING

摘要

An illustrative embodiment of a computer-implemented process for partitioning a crawling space computes an event identifier for each event in the set of events to form an identified set of events, segments the identified set of events into a number of partitions, assigns a partition to each node in a set of nodes and executes each event in each assigned partition by a respective node. Responsive to a determination that a new state is discovered, other nodes are notified of the new state, wherein information associated with the new state is added to a respective assigned set of event IDs at each node. Responsive to a determination that no more notifications exist, the computer-implemented process determines whether more events to process exist and responsive to a determination that no more events to process exist, the computer-implemented process terminates.