METHOD AND SYSTEM FOR DYNAMICALLY MANAGING BIG DATA IN HIERARCHICAL CLOUD STORAGE CLASSES TO IMPROVE DATA STORING AND PROCESSING COST EFFICIENCY

摘要

A system and method for autonomic data storage and movement for big data analytics. A cost, such as storing cost and a processing cost are calculated for received data. The processing type associated with the received data is determined in response to the calculated costs. The received data is classified as one of a set of hierarchical storage classes based upon the determined processing type. The hierarchical storage classes include no data store, memory, HDFS, database, disk archive, external clouds, and data removal. The received data is then stored in the storage location associated with that class. In the event that insufficient capacity is available in the location, the priority of the received data and the priority of previously stored data is determined and compared. The priority is calculated based on potential usage, privacy, estimated cost, frequency of usages and the age of data. The lower priority data is then moved to the next lower hierarchical class for storage.

主权项

1. A method for autonomic data storage and movement, comprising:
calculating at least one cost associated with received data; responsive to at least one calculated cost, determining a processing type associated with the received data; classifying the received data as one of a set of hierarchical storage classes in accordance with the determined processing type; identifying a storage location associated with the one of the set of hierarchical storage classes in which the received data was classified; and storing the received data in the identified storage location, wherein at least one of the calculating, classifying, identifying, and storing is performed with a computer processor.