System and method for automating data warehousing processes

摘要

A system and computer-implemented method for automating data warehousing processes is provided. The system comprises a code generator configured to generate codes for Extract, Transform and Load (ETL) tools, wherein the codes facilitate the ETL tools in extracting, transforming and loading data read from data sources. The system further comprises a code reviewer configured to review and analyze the generated codes. Furthermore, the system comprises a data migration module configured to facilitate migrating the data read from the data sources to one or more data warehouses. Also, the system comprises a data generator configured to mask the data read from the data sources to generate processed data. In addition, the system comprises a Data Warehouse Quality Assurance module configured to facilitate testing the read and the processed data. The system further comprises a reporting module configured to provide status reports on the data warehousing processes.

主权项

1. A computer system for automating one or more data warehousing processes, the computer system comprising a processor and a memory, the computer system further comprising:
a code generator connected to one or more Extract, Transform and Load (ETL) tools using adapters and connectors, the code generator is configured to generate, using the processor, codes that facilitate the one or more ETL tools to extract, transform and load data read by a data acquisition module from one or more data sources, wherein the code generator is connected to the one or more Extract, Transform and Load (ETL) tools based on one or more connection parameters received from one or more users; a code reviewer configured to review and analyze, using the processor, the generated codes, wherein the review and analysis of the generated codes comprises identifying unused variables in the generated codes that clog the memory; a data migration module configured to facilitate, using the processor, migrating the data read from the one or more data sources to one or more data warehouses, wherein the reviewed generate codes facilitate in migrating the read data to the one or more data warehouses; a data generator configured to mask, using the processor, the data read from the one or more data sources to generate processed data for testing; a Data Warehouse Quality Assurance (DW QA) module configured to facilitate, using the processor, testing at least one of: the read data and the processed data; and a reporting module configured, using the processor, to provide one or more status reports on the one or more data warehousing processes, wherein a dictionary matcher facilitates the reporting module in generating a report on outliers in the read data by identifying and highlighting outliers in the read data.