Avoidance of intermediate data skew in a massive parallel processing environment

摘要

A computer-implemented method for minimizing join operation processing time within a database system based on estimated joined table spread of the database system has been provided. The computer-implemented method includes estimating value distribution of data in a joined table, wherein the joined table is a result of join operation between two instances of tables of a database system. The computer-implemented method further includes determining boundaries for partitioning at least one range of attributes of the estimated value distribution, wherein the boundaries for partitioning at least one range of attributes of the estimated value distribution corresponds to a same number of rows of the joined table. The computer-implemented method further includes determining at least one assignment of the determined partition of the at least one range of attributes to processing units of the database system.

主权项

1. A computer system for minimizing join operation processing time within a database system based on estimated joined table spread of the database system, the computer system comprising:
one or more processors, one or more computer-readable memories, one or more computer-readable tangible storage devices and program instructions which are stored on at least one of the one or more storage devices for execution by at least one of the one or more processors via at least one of the one or more memories, the program instructions comprising: program instructions to estimate value distribution of data in a joined table, wherein the joined table is a result of a join operation between two instances of tables of a database system and the estimated value distribution data is based on a count of number of rows with a particular value of the two instances of the tables of the join operation and calculation of a number of rows of the joined table; program instructions to determine boundaries for partitioning at least one range of attributes of the estimated value distribution of data, wherein the boundaries for partitioning is based on a density function distribution of the joined table, and wherein the density distribution function represents a spread of data within a data column of the joined table, the joined table being a result of a join operation between two instances of tables of the database system;
program instructions to determine at least one assignment of the determined partition of the at least one range of attributes to processing units of the database system;program instructions to provide parameterized functions that represent the estimated value distribution;program instructions to determine the total number of rows in the join table;program instructions to determine a set of estimates for the number of rows in the join table in a first set of consequent common attribute value ranges based on the parametrized functions;program instructions to determine a parametrized function representing the value distribution of the common attribute in the join table by solving a first equation system involving definite integrals of over the first set of consequent attribute ranges and the set of estimates for numbers of rows; andprogram instructions to determine a second set of consequent common attribute value ranges by solving a second equation system involving definite integrals of over the second set of consequent common attribute value ranges, each definite integral equalling the total number of rows in the join table divided by the number of processing nodes.