Interdistinct operator

摘要

A computer-implemented system and method for performing distinct operations on multiple tables of shared memory of parallel computing environments are disclosed. A distinct operation is executed on each table of a plurality of tables, each distinct operation eliminating duplicate data from each table, the executing creating a hierarchy of table pairs and distinct results, the distinct results comprising a reduced row set for each table. Duplicates on each reduced row set are detected to complete the distinct operation on the plurality of tables.

主权项

1. A computer-implemented method comprising:
executing a distinct operation on each table of a plurality of tables shared by a plurality of processing units in a parallel computing environment, each table comprising a plurality of rows and a plurality of columns, each column comprising values of a corresponding attribute of a plurality of attributes, each distinct operation eliminating duplicate data from the plurality of columns in each table, an order in which a column of each table is selected for removal of the duplicate data being based on a discrimination value of an attribute specific to the column and on a dependency metric characterizing dependency between the attribute specific to the column and one or more attributes specific to other one or more columns from which duplicate data has previously been removed, the executing creating a hierarchy of table pairs and distinct results, the distinct results comprising a reduced row set for each table, at least one table pair of the table pairs having a maximum row number difference between two corresponding tables of the at least one table pair; and detecting duplicates on each reduced row set to complete the distinct operation on the plurality of tables, the executing and detecting being performed by one or more processors.