Hybrid data management system and method for managing large, varying datasets

摘要

A hybrid data management/storage system is provided which includes two or more integrated or connected data management systems. An external application and/or user interacts with the hybrid data management/storage system using a unified interface. Incoming raw data may be directed to be stored in any of a plurality of data management systems based on the incoming data object having one or more of a number of predefined characteristics, including for example size and/or data type. Metadata corresponding to all incoming data objects may be stored in a particular data store, regardless of whether the incoming object's raw data is stored in a different one of the plurality of data stores.

主权项

1. A method for storing an incoming data object including metadata having first preferred predefined characteristics and raw data having second preferred predefined characteristics, the method comprising the acts of:
providing a hybrid data storage management system comprising:
a plurality of data stores including at least a first data store and a second data store different from the first data store,wherein each of the plurality of data stores is associated with a preferred data store type corresponding to a type of data store whose storage method permits the associated data store to operate more efficiently on data having preferred predefined characteristics associated with the data store type than on data not having the preferred predefined characteristics,wherein the first data store is a first preferred data store having a first preferred data store type corresponding to first preferred predefined characteristics, and the second data store is a second preferred data store having a second preferred data store type corresponding to second preferred predefined characteristics; receiving, by the hybrid data management system, from an external source, a write request for the incoming data object; determining, by the hybrid data management system, that the metadata and the raw data of the incoming data object have the first and second preferred predefined characteristics, respectively; identifying, by the hybrid data management system, the first and second preferred data store types corresponding to the metadata and the raw data, respectively, based on the first and second preferred predefined characteristics; routing, by the hybrid data management system, the raw data to the second data store for storage therein based on the raw data having been identified as corresponding to the second data store type; and storing, by the hybrid data management system, the metadata in the first data store based on the metadata having been identified as corresponding to the first data store type, and without regard to which of the plurality of data stores is the second preferred data store corresponding to the raw data of the incoming data object, the metadata including associative information linking the metadata with the corresponding raw data in the second data store, wherein the first data store and the second data store utilize different storage methods for at least one of data replication, retention and distribution, such that, by virtue of a first data store storage method, the first data store operates more efficiently on the metadata than the second data store would, and, by virtue of a second data store storage method, the second data store operates more efficiently on the raw data than the first data store would.