System and method for synchronization of haptic data and media data

摘要

Embodiments of the present invention relate to computer systems for transmitting or receiving and playing a plurality of data streams, where at least one of the data streams includes video data packets and/or audio data packets and at least one other data stream includes further data packets that are synchronized with the video and/or audio data packets. In particular embodiments, the further data stream includes haptic data packets that include haptic command data generated in real time with the video and/or audio data streams, where the haptic command data is provided to a haptic output device configured to replicate or approximate sensations through the output of one or more mechanical forces in synchronization with the playing of video and/or audio from the video and/or audio data streams.

主权项

1. An apparatus, comprising at least one processor and at least one memory including computer program code, the at least one memory and the computer program code configured to, with the at least one processor, cause the apparatus to perform at least the following:
receive, from a source computer system, a plurality of time stamped data streams including an additional data stream and at least one of an audio data stream or a video data stream, each data stream having data packets that are time stamped, wherein the audio data stream includes audio packets that contain timestamped audio information for an audio device and the video data stream includes video packets that contain timestamped video information for a video display device, and wherein the additional data stream includes command data packets that contain timestamped command data for an additional device; determine a time stamp of a command data packet received in the additional data stream, wherein the timestamp corresponds to a time when the command data was received by the source computer; determine a time stamp of at least one of an audio data packet received in the audio data stream or a video packet received in the video data stream, wherein at least one of: (a) the timestamp of the audio packet corresponds to a time when the audio information was received by the source computer, or (b) the timestamp of the video packet corresponds to a time when the video information was received by the source computer; time align the timestamped command data with at least one of the timestamped audio information or the timestamped video information in at least one buffer, according to the timestamp determined for the command data packet received in the additional data stream and the timestamp determined for the at least one of the audio packet or the video packet received in at least one of the audio data stream or the video data stream; and generate output signals for controlling a plurality of output devices to provide time aligned output according to the command data in data packets of the additional data stream, and the at least one of the audio information in the audio data stream or the video information in the video data stream that is time aligned with the command data; wherein at least one of (a) the audio packet from the audio data stream is adapted to control an audio device, at a time based on the timestamp of the audio packet, time aligned with the timestamp of the data packet from the additional data stream, or (b) the video packet from the video data stream is adapted to control a video display device, at a time based on the timestamp of the video packet, time aligned with the timestamp of the data packet from the additional data stream.