Infrastructure enabling intelligent execution and crawling of a web application

摘要

In one embodiment, a method includes accessing a structured document of a network application, processing the structured document to generate a model representation of the structured document, tracking one or more interactions occurring during the processing of the structured document, the one or more interactions including one or more outgoing requests transmitted by the one or more computing systems or incoming responses received by the one or more computing systems, and generating a behavior model of the web application based on one or more of the interactions.

主权项

1. A method comprising:
accessing, by a web crawler executing on one or more computing systems associated with a social-networking system, a structured document of a network application, the structured document comprising structural information and content comprising one or more embedded scripts and one or more resources or identifiers for the resources; executing, by the web crawler executing on the one or more computing systems, at least some of the content of the structured document; processing, by the computing systems, the structured document to generate a model representation of the structured document; tracking, by the computing systems, one or more interactions resulting from the web crawler's execution of at least some of the content, the interactions comprising one or more outgoing requests sent by one or more of the computing systems or incoming responses received by one or more of the computing systems from one or more third-party servers; creating, by the computing systems, a behavior model of the network application based on one or more of the interactions resulting from the web crawler's execution of at least some of the content, the behavior model comprising a first log of outgoing HTTP requests generated by the network application when the content is executed; creating, by the computing systems, a second log that comprises an identification of one or more network resources ascertained by filtering the first log; comparing, by the computing systems, one or more of the network resources identified in the second log to a list comprising an identification of one or more rogue network resources; by the computing systems, determining, based on the comparison, whether the network application meets one or more requirements of the social-networking system, wherein the one or more requirements comprise avoiding interaction with any of the rogue network resources.