TABLE LEVEL DISTRIBUTED DATABASE SYSTEM FOR BIG DATA STORAGE AND QUERY

摘要

A method of managing on-line user data in a table-level distributed database system is disclosed. At a loading interface, a load request from a client device specifying a new or updated data table, which has been populated in a distributed file system, is received. The load request also specifies a location in the distributed file system and a schema of the data table. The loading interface creates new records in a metadata store, and each new record specifies a mapping between the data table and a particular instance that is to load such data table. Each record also specifies the location and schema for such data table. Based on the new records, each instance also loads the data table. Each instance publishes, in the metadata store, load status specifying that loading of the data table is complete so that such data table can be queried by a query client.

主权项

1. A table-level database system for managing on-line user data, the system comprising:
a metadata storage system for holding metadata records; a loading interface for receiving from a client device a plurality of load requests having location and schema information for periodically generated or updated data tables that have been populated in a distributed file system, wherein the loading interface is further configured to store in the metadata storage system, in response to each load request, a metadata record mapping the loading request's schema and location information to identifying information for each one or more selected ones of a plurality of distributed table storage instances into which to upload the data table of such load request; the plurality of storage instances each being configured for and having buffers for loading each identified one of the data tables that is mapped to identifying information for the storage instance as specified in a specific metadata record into the storage instance from the distributed file system based on periodically obtaining, from the metadata storage system, a location and schema information for each identified one of the data tables, each storage instance being further configured for outputting results via the buffers in response to queries on one or more data tables that are loaded into such storage instances; a query interface for receiving, from a client device, client queries on data tables and forwarding such client queries to obtain results from one or more of the storage instances based on metadata records stored in the metadata storage system.