Translating text to, merging, and optimizing graphical user interface tasks

摘要

A system and method that enables a plurality of lay users to collaborate on automating computer tasks is disclosed. In one embodiment, the system automatically performs these tasks, rather than just documenting how to perform them. The system allows a database of solutions to be built for every important computer task. A key characteristic of this system is that users contribute to this database by simply performing the task. The system records the graphical user interface (GUI) actions as the user performs the task. It aggregates GUI traces from multiple users into a canonical sequence of GUI actions parameterized by user-environment that will successfully accomplish the task on a variety of different configurations. A classifier is used to predict which steps are likely to be misinterpreted and requests human intervention to properly perform them. This process can be done iteratively until the translation is believed to be correct.

主权项

1. A method of generating a set of steps to perform a computer based task, comprising:
a. having one or more users perform said task using a respective computing device; b. delivering one or more traces from said users performing said task to a processing unit, wherein each of said traces represent GUI actions performed on a user interface of said respective computing device, wherein said user interface comprises a series of views, and each of said views comprises a plurality of widgets, each widget configured to accept user input, and wherein said traces comprise a list of GUI actions performed on said plurality of widgets in said series of views; and c. using said processing unit to execute a set of rules on said traces to generate said set of steps, wherein said set of rules comprises: mapping each GUI action to an abstract action based on the GUI action's effect on the state of the system, by:
determining whether the GUI action causes a change or a pending change to the state of the system, and if it is determined that the GUI action causes a change or pending change to the state of the system, categorizing the GUI action as a state action; anddetermining whether the GUI action causes a change in the plurality of visible widgets, and if it is determined that the GUI action causes a change in the plurality of visible widgets, categorizing the GUI action as a navigation action;optimizing said abstract actions wherein optimizing said abstract actions comprises comparing the abstract actions from said one or more traces and differentiating between differences due to user error and those due to configuration differences based on subsequent views; andgenerating said set of steps from said optimized abstract actions.