Data Archive Vault in Big Data Platform

摘要

Embodiments relate to data archiving utilizing an existing big data platform (e.g., HADOOP) as a cost-effective target infrastructure for storage. Particular embodiments construct a logical structure (hereafter, “vault”) in the big data platform so that a source, type, and context of the data is maintained, and metadata can be added to aid searching for snapshots according to a given time, version, and other considerations. A vaulting process transforms relationally stored data in an object view to allow for object-based retrieval or object-wise operations (such as destruction due to legal data privacy reasons), and provide references to also store unstructured data (e.g., sensor data, documents, streams) as attachments. A legacy archive extractor provides extraction services for existing archives, so that extracted information is stored in the same vault. This allows for cross queries over legacy data and data from other sources, facilitating the application of new analysis techniques by data scientists.

主权项

1. A computer-implemented method comprising:
an engine of a big data platform receiving from an application layer, a first input comprising a plurality of fields organized in a first data structure; the engine receiving from the application layer, context information relevant to the first data structure; and the engine storing in a vault of the big data platform, values of the plurality of fields and the context information organized as a second data structure different from the first data structure.