SYSTEMS AND METHODS FOR DATA VALIDATION AND PROCESSING USING METADATA

摘要

A system receives a source and a metadata layer that describes the source. The source may comprise source records with fields containing source data, and the metadata layer may include metadata comprising at least one of a field data type, a field data length, a field description, or a record length. The processor may further validate the metadata layer against the source and write results to a log. The processor may further be configured for transforming the source records into transformed records for a load ready file. The processor may further balance a number of records in the source against a number of transformed records in the load ready file to generate a transformation failure rate.

主权项

1. A method comprising:
receiving, by a processor, a source and a metadata layer that describes the source, wherein the source comprises source records with fields containing source data, wherein the metadata layer includes metadata comprising at least one of a field data type, a field data length, a field description, or a record length; validating, by the processor, the metadata layer against the source; writing, by the processor, results to a log; transforming, by the processor, the source records into transformed records for a load ready file, wherein the transformed records comprise a derived field in response to a transformation applied to the fields containing the source data, wherein a bad record is written to the log in response to a failed transformation; balancing, by the processor, a number of records in the source against a number of transformed records in the load ready file to generate a transformation failure rate; deciding, by the processor, a state of the transforming the source records in response to the transformation failure rate and a predetermined acceptable failure rate; and outputting, by the processor, the load ready file.