Interactive mixed reality system and uses thereof

摘要

An interactive mixed reality simulator is provided that includes a virtual 3D model of internal or hidden features of an object; a physical model or object being interacted with; and a tracked instrument used to interact with the physical object. The tracked instrument can be used to simulate or visualize interactions with internal features of the physical object represented by the physical model. In certain embodiments, one or more of the internal features can be present in the physical model. In another embodiment, some internal features do not have a physical presence within the physical model.

主权项

1. An interactive mixed reality system, comprising:
a physical model of an object, wherein the physical model is penetrable through an external or internal surface at any location and at any orientation; one or more tracked instruments for interacting with the physical model; a tracking system for tracking a position and an orientation of the one or more tracked instruments relative to the physical model; a virtual model, stored in a memory, comprising one or more 3-D virtual representations of features of the object, the features comprising internal or hidden features of the object, the one or more 3-D virtual representations comprising abstract or concrete representations; a simulation module that: accesses the virtual model stored in the memory; receives input from the tracking module regarding the position and orientation of the one or more tracked instruments relative to the physical model; determines, based on the input from the tracking system, a corresponding interaction between the one or more tracked instruments and the one or more 3-D virtual representations of the virtual model; determines a consequence of the corresponding interaction; and renders, to a display, a virtual representation comprising a particular set of the one or more 3-D virtual representations, the corresponding interaction, and the consequence of the corresponding interaction, wherein the corresponding interaction is a physical interaction between the one or more tracked instruments within the physical model and the internal or hidden features of the object in the virtual model.