SCALABLE INDEX BUILD TECHNIQUES FOR COLUMN STORES

摘要

Architecture that includes an index creation algorithm that utilizes available resources and dynamically adjusts to successfully scale with increased resources and be able to do so for any data distribution. The resources can be processing resources, memory, and/or input/output, for example. A finer level of granularity, called a segment, is utilized to process tuples in a partition while creating an index. The segment also aligns with compression techniques for the index. By choosing an appropriate size for a segment and using load balancing the overall time for index creation can be reduced. Each segment can then be processed by a single thread thereby limiting segment skew. Skew is further limited by breaking down the work done by a thread into parallelizable stages.

主权项

1. A computing device, comprising:
a memory and a processor that are respectively configured to store and execute instructions, including instructions for causing the computing device to perform operations, the operations comprising: selecting a segment size for partitions corresponding to a query execution; building an index based on the segment size; and processing segments according to the index based on a resource optimization determined at a query execution time.