Data integration using automated data processing based on target metadata

摘要

Approaches for data integration between multiple IT sources using automated data processing based on target metadata are provided. Specifically, an integration component is configured to load a mapped data set into a table with delta processing based on a configuration object containing, e.g., the source data location, target table name, and source to target mapping. The integration component uses the target metadata to validate data, identify changes, generate the necessary database programming language (e.g., structured query language (SQL)), and run the database programming language with data binding to perform the actual data updates. The integration component leverages the data target metadata to automate the processing of source data, thus providing a way to validate the data, and identify delta changes at the field level between the source and target. This significantly reduces the overall development effort, while providing consistency in record handling and error reporting.

主权项

1. A method for data integration using automated data processing based on target metadata, the method comprising the computer-implemented steps of:
defining an intermediate data table built from target metadata of a target source destination containing a set of data objects, the defining the intermediate data table comprising: generating a table definition built from target metadata; generating a column definition built from target metadata; and generating an object representation of a record based on the table definition and the column definition; loading a mapped data set containing a set of source data objects from a source into the intermediate data table based on a configuration object, and the object representation; sorting the set of data objects built from target metadata; sorting and validating source data in the intermediate data table using the target metadata; comparing the set of data objects to the sorted and validated source data to identify differences using the target metadata; generating a set of programming language statements to map the differences in the source data based on the target metadata using the target metadata; and loading the mapped source data to a target storage destination using the set of programming language statements.