ETL DIAGNOSTICS

摘要

A cloud-based ETL system provides error detection, error correction and reporting of data integration flows hosted by cloud services. Categories of errors are identified using one or more checks at different points of a data integration flow and one or more actions selected based at least in part on the error category. A determination can be made whether the error category is fault tolerant and one or more actions can be selected based at least in part on the error fault tolerance to correct the error, restart a flow, or generate a notification assisting a user to correct the error.

主权项

1. A method, comprising:
identifying, by a computer system, an error in a cloud-based data integration scenario having data flowing from a source to a target; determining, by the computer system, whether the error is correctable; performing, by the computer system, one or more actions that correct the error based at least in part on a category of the error when the error is correctable; determining, by the computer system, whether the error is fault tolerant based at least in part on the category of the error when the error is not correctable; processing, by the computer system, a first portion of the data in a first manner that rejects the data and a second portion of the data in a second manner that transforms the data when the error is fault tolerant; and generating, by the computer system, one or more notifications relevant to the error.