TRANSFORMING AND LOADING DATA UTILIZING IN-MEMORY PROCESSING

摘要

A system includes at least one processor and processes an ETL job. The system analyzes a specification of the ETL job including one or more functional expressions to load data from one or more source data stores, process the data in memory, and store the processed data to one or more target data stores. One or more data flows are produced from the specification based on the one or more functional expressions. The one or more data flows utilize in-memory distributed data sets generated to accommodate parallel processing for loading and processing the data. The one or more data flows are optimized to assign operations to be performed on the one or more source data stores. The optimized data flows are executed to load the data to the one or more target data stores in accordance with the specification. Present invention embodiments further include methods and computer program products.

主权项

1. A method of processing an Extract, Transform, Load (ETL) job comprising:
analyzing a specification of the ETL job including one or more functional expressions to load data from one or more source data stores, process the data in memory, and store the processed data to one or more target data stores; producing one or more data flows from the specification based on the one or more functional expressions, wherein the one or more data flows utilize in-memory distributed data sets generated to accommodate parallel processing for loading and processing the data; optimizing the one or more data flows to assign operations to be performed on the one or more source data stores; and executing the optimized data flows to load the data to the one or more target data stores in accordance with the specification.