User interface remoting through video encoding techniques

摘要

Methods, techniques, and systems for user interface remoting using video streaming techniques are provided. Example embodiments provide User Interface Remoting and Optimization System (“UIROS”), which enables the efficient remoting of pixel-oriented user interfaces on behalf of their guests using generic video streaming techniques, such as H.264, to send compressed user interface image information in the form of video frame encoded bitstreams. In one embodiment, the UIROS comprises server side support including a UI remoting server, a video encoder, and rendering support and client side support including a UI remoting client, a video decoder, and a display. These components cooperate to implement optimized UI remoting that is bandwidth efficient, low latency and CPU efficient.

主权项

1. A method in a server computing system for remoting a user interface to be rendered on a client device, comprising:
during a motion estimation process of a generic video encoder, causing the generic video encoder to:
perform pattern matching on a graphics command stream of a quest operating system of a virtual machine on the server computing system in order to detect a move, scroll or expose region event;when the move, scroll or expose region event is detected, determine, prior to causing an updated portion of the user interface to be rendered on a display screen associated with the client device, a set of pixels that can be reused in the updated portion of the user interface and identify a motion vector caused by the move, scroll or expose region event, wherein the motion vector indicates new locations for the set of pixels in the updated portion of the user interface;using a generic video encoding protocol supported by a web browser of the client device, generate and encode a video-based representation of the updated portion of the user interface that indicates reuse of the determined set of pixels that can be reused in the updated portion of the user interface and the motion vector that indicates the new locations; and sending the generated and encoded video-based representation of the updated portion of the user interface to the web browser on the client device in a manner that causes a decoder for the video encoding protocol on the web browser of the client device to decode and render the updated portion of the user interface on the display screen associated with the client device by using the set of pixels cached on the client device and the motion vector caused by the move, scroll or expose region event.