System and method for flexible data transfer

摘要

A method and system for flexibly transferring data from one or more data sources to one or more data destinations within an information network where each of the one or more data sources have data in a particular source format and each of the one or more data destinations have data in the same or another particular destination format using a parameter database that includes parameters to control the transfer of data, a scheduler that initiates the transfer of data, and a data loader in communications with the parameter database and scheduler that, upon initiation by the scheduler, extracts data from the one or more data sources, manipulates the extracted source data into one or more destination formats associated with the one or more data destinations, and inserts the data into one or more data destinations according to the parameters within the parameter database.

主权项

1. A data loading method, comprising:
assigning, by a data loader, different batch numbers to different groups of data loading operations; launching concurrently a first group of threads corresponding to a first group of data loading operations having a first batch number; awaiting completion of the first group of threads; and responsive to detecting completion of the first group of threads, launching concurrently a second group of threads corresponding to a second group of data loading operations having a second batch number; wherein launching the first group of threads includes launching a plurality of threads concurrently wherein each of the plurality of threads is associated with a corresponding one of the first group of data loading operations.