Technique for coordinating the distributed, parallel crawling of interactive client-server applications

摘要

An electronic device includes a memory and a processor coupled to the memory. The memory contains a master state graph. The master state graph includes information regarding the operation of interactive client-server application. The processor is configured to send a first job to a first worker node, send a second job to a second worker node, receive results of crawling the interactive client-server application, and integrate results of crawling the interactive client-server application into the master state graph. The first job includes crawling instructions for crawling a first portion of an interactive client-server application. The second job includes crawling instructions for crawling a second portion of the interactive client-server application. The first worker node and second worker node crawl the interactive client-server application in parallel.

主权项

1. An electronic device, comprising:
a memory, the memory containing a master state graph, the master state graph comprising information regarding the operation of an interactive client-server application; and a processor coupled to the memory; wherein the processor is configured to:
send a first job to a first worker node, the first job comprising crawling instructions for crawling a first portion of an interactive client-server application;send a second job to a second worker node, the second job comprising crawling instructions for crawling a second portion of the interactive client-server application, the first worker node and second worker node crawling the interactive client-server application in parallel;receive results of crawling the interactive client-server application, the results including a compressed local state graph including information of successive states determined while crawling;decompress the compressed local state graph yielding a full representation of the successive states crawled; andintegrate results of crawling the interactive client-server application into the master state graph.