A COGNITIVE CONTROL FRAMEWORK FOR AUTOMATIC CONTROL OF APPLICATION PROGRAMS EXPOSING A GRAPHICAL USER INTERFACE

摘要

A cognitive control framework system for automatically controlling execution of an application program having a graphical user interface includes a recording component, an execution scenario script, and a playback component. The recording component is adapted to capture user input data and images displayed by the graphical user interface during a recording phase of execution of the application program, and to analyze the captured user input data and displayed images to generate an execution scenario (script) during the recording phase. The execution scenario may be written in a selected high level language (e.g., XML). The playback component is adapted to generate simulated user input data based on the execution scenario during a playback phase of execution of the application program, to input the simulated user input data to the application program, to perform image analysis on images displayed by the graphical user interface as a result of processing the simulated user input data during the playback phase and captured displayed images from the recording phase; and to automatically control execution of the application program based at least in part on the image analysis.