SYSTEM AND METHOD FOR TRANSFERRING DATA BETWEEN RDBMS AND BIG DATA PLATFORM

摘要

A system for transferring data from a Relational Database Management System (“RDBMS”) to a big data platform and methods for making and using the same. The system can acquire a partitioning execution scheme of a selected table from the RDBMS and submitting partitioned queries from the big data platform to each mapper of partitions. The partitioned queries are generated based on the partitioning execution scheme. The partitioning execution scheme can be acquired by submitting a query explain request to an optimizer of the RDBMS to generating a parallel query plan. The partitioning execution scheme can also be acquired by querying statistics from a statistics catalog of the RDBMS or by user inputs. The system can use RDBMS capabilities and statistics for parallel data fetching. Thereby, the system can increase efficiency of the fetching and can avoid straggling when target data is not evenly distributed and can avoid table query-in-serial.

主权项

1. A method for transferring data from a Relational Database Management System (“RDBMS”) to a big data platform, comprising:
acquiring a partitioned execution scheme of a selected table from the RDBMS; generating partitioned queries, with partition mappers, based on said acquired partitioned execution scheme; and submitting partitioned queries from the big data platform via the partition mappers.