Systems and methods for implementing pixel-based reverse engineering of interface structure

摘要

Systems and methods for implementing real-time pixel-based reverse engineering of interface structures are disclosed herein. The system uses the pixels representing widgets and content of a user interface to analyze, interpret, and/or enhance the interface without requiring cooperation from the interface. The system captures pixel data and reverse engineers the composition and structure of the interface from the captured pixel data. Using the reverse engineered interface, generated based on the captured pixel data, the facility can identify and record user interactions with the interface, modify the display of the interface to meet user requirements, preferences, or customization options, or enhance the interface to provide functionality not otherwise available, etc. The system can perform these processes across a variety of interfaces regardless of their underlying implementation and without any modifications to the interface.

主权项

1. A method, performed by a computer having a memory and a processor, for interpreting a user interface, the method comprising:
capturing at least a portion of pixel data of a source window associated with the user interface; with a processor, identifying areas of the captured pixel data that at least substantially match features associated with at least one widget prototype of a prototype library comprising a plurality of widget prototypes, wherein each feature comprises a plurality of pixels; and with a processor, identifying a user interface widget within the captured pixel data at least in part by determining whether a spatial relationship between at least two of the identified areas is consistent with at least one widget prototype of the prototype library,
wherein a first widget prototype of the prototype library includes,
a plurality of features, wherein the plurality of features is representative of a first type of user interface widget, andat least one constraint on a spatial relationship between at least two features of the first widget prototype, andwherein the identifying the user interface widget within the captured pixel data is based at least in part on:
matching pixels of a plurality of features of the first widget prototype to pixels of the captured pixel data to identify a plurality of features within the captured pixel data corresponding to the user interface widget within the captured pixel data,selecting a first feature of the identified plurality of features within the captured pixel data,selecting a second feature of the identified plurality of features within the captured pixel data, anddetermining whether the first feature of the identified plurality of features within the captured pixel data and the second feature of the identified plurality of features within the captured pixel data satisfy the at least one constraint of the first widget prototype.