Hierarchy of servers for query processing of column chunks in a distributed column chunk data store

摘要

An improved system and method for query processing in a distributed column chunk data store is provided. A distributed column chunk data store may be provided by multiple storage servers operably coupled to a network. A storage server provided may include a database engine for partitioning a data table into the column chunks for distributing across multiple storage servers, a storage shared memory for storing the column chunks during processing of semantic operations performed on the column chunks, and a storage services manager for striping column chunks of a partitioned data table across multiple storage servers. Query processing may be performed by storage servers or query processing servers operably coupled by a network to storage servers in the column chunk data store. To do so, a hierarchy of servers may be dynamically determined to process execution steps of a query transformed for distributed processing.

主权项

1. A method implemented on at least one machine, each of which has at least one processor, storage, and a communication platform, comprising:
accessing, at a first query processing server, a query for data of a table comprising a plurality of columns and a plurality of rows, wherein:
the table has been partitioned into a plurality of column chunks by independently partitioning each column of the table into two or more column chunks based on a predetermined partitioning policy, the predetermined partitioning policy comprising information related to the number of column chunks into which the each column of the table should be partitioned;a plurality of parity column chunks are created, each parity column chunk being created based on two or more column chunks of the plurality of column chunks; andthe plurality of column chunks and the plurality of parity column chunks have been distributed and stored across a plurality of storage servers; transforming, by the first query processing server, the query into a plurality of sub-queries; distributing, by the first query processing server, the plurality of sub-queries to a plurality of second query processing servers; receiving, at the first query processing server, the data from the plurality of second query processing servers; and transmitting, by the first query processing server, the data in response to the query.