Static-Analysis-Assisted Dynamic Application Crawling Architecture

摘要

A system for automated acquisition of content from an application includes a static analysis module. Using static analysis (analyzing code of the application independent of executing the code), the static analysis module identifies application state templates of the application and identifies user interface elements that, when actuated by a user, transition between states instantiated from different ones of the identified state templates. A graph creation module creates a graph data structure including nodes corresponding to the identified state templates and edges connecting the nodes. The edges are transitions corresponding to the identified user interface elements. A link extractor module executes the application and determines and records paths to states within the executing application according to the graph data structure. A scraper module, within an executing instance of the application, extracts text and metadata from states corresponding to endpoints of the recorded paths.

主权项

1. A system for automated acquisition of content from an application, the system comprising:
a static analysis module configured to, using static analysis:
identify application state templates of the application andidentify user interface elements that, when actuated by a user, are configured to transition between states instantiated from different ones of the identified application state templates,wherein the static analysis includes analyzing code of the application independent of executing the code; a graph creation module configured to create a graph data structure including nodes and edges connecting the nodes, wherein the nodes correspond to the identified application state templates, and wherein the edges correspond to transitions between the identified application state templates corresponding to the identified user interface elements; a link extractor module configured to:
execute the application anddetermine and record paths to application states within the executing application according to the graph data structure; and a scraper module configured to, within an executing instance of the application, extract text and metadata from application states corresponding to endpoints of the recorded paths, wherein information based on the extracted text and metadata is stored in a data store.