Systems and Methods for Providing Metadata Aware Background Caching in Data Analysis

摘要

In general, the present invention is directed to systems and corresponding methods for providing metadata aware background caching amongst various tables in data processing systems, the system configured to process either an original copy of data stored or data stored in derived tables in one or more data stores, the system including: a query optimization module, a catalog module, and a dataset manager. Each of the query optimization module, catalog module, and dataset manager may be communicatively connected to the original copy of data and the derived tables in one or more data stores. The query optimization module configured to conduct queries against data stored in the original copy of data or in the derived tables; the catalog module configured to register tables of data across various types and formats of data stores; and the dataset manager configured to maintain the freshness of the data in the derived tables.

主权项

1. A system for providing metadata aware background caching amongst various tables in data processing systems, the system configured to process either an original copy of data stored in a first format or data stored in one or more derived tables in one or more data stores, the system comprising:
a query optimization module, the query optimization module communicatively connected to the original copy of data, the one or more derived tables, and a catalog module, the query optimization module configured to conduct queries against data stored in the original copy of data and/or in the one or more derived tables; a catalog module, communicatively connected to the original copy of the data and the one or more derived tables, the catalog module in further communication with the query optimizer and a dataset manager, the catalog module configured to register tables of data across various types and formats of data stores; a dataset manager, communicatively connected to the original copy of the data, the one or more derived tables, and the catalog module, the dataset manager configured to maintain the freshness of the data in the one or more derived tables.