END TO END VALIDATION OF DATA TRANSFORMATION ACCURACY

摘要

Data is validated as it travels through the different nodes of a data pipeline. Instead of having to wait to validate the data when the data reaches an end of the data pipeline, each node in the pipeline may validate the data. Different methods may be used to validate the data. For example, each node may determine metadata about the received data and/or the transformed data. This metadata may be used to determine if the node is receiving the same amount of data as it usually receives, whether the data is in a same format, and the like. A timing of the data through one or more of the nodes may also be used in determining when the data is valid. When a problem is detected at any of the nodes in the pipeline, a report may be sent to one or more users.

主权项

1. A method for validating data at a node in a data pipeline that includes a plurality of nodes, comprising:
receiving data at the node; determining current metadata about the data that includes determining characteristics of the data that includes a size of the data; comparing the current metadata to other metadata related to the data to determine a validity of the data that is based on a difference between the current metadata and the other metadata; and sending a report that includes information relating to the validity of the data.