SYSTEM AND METHOD FOR DECOMPOSITION OF CODE GENERATION INTO SEPARATE PHYSICAL UNITS THOUGH EXECUTION UNITS

摘要

In various embodiments, a data integration system is disclosed which enables users to create a logical design which is platform and technology independent. The user can create a logical design that defines, at a high level, how a user wants data to flow between sources and targets. The tool can analyze the logical design, in view of the user's infrastructure, and create a physical design. The logical design can include a plurality of components corresponding to each source and target in the design, as well as operations such as joins or filters, and access points. Each component when transferred to the physical design generates code to perform operations on the data. Depending on the underlying technology (e.g., SQL Server, Oracle, Hadoop, etc.) and the language used (SQL, pig, etc.) the code generated by each component may be different.

主权项

1. A method of generating a physical design from a logical design for data integration, comprising:
receiving a logical design of a data integration mapping; applying a plurality of rules to the logical design; generating a default physical design comprising a plurality of execution units based on the logical design; and grouping the plurality of execution units into one or more execution unit groups, wherein each execution unit group includes independent execution units which can execute in parallel.