Resilient data processing pipeline architecture

摘要

A private cloud-based processing pipeline apparatus and method for its use is disclosed. A first load balancer directs data packets to one of a plurality of collectors. The collectors authenticate the data packet. Then a second load balancer receives the data packet from the collector and to direct the data packet to one of a plurality of extract transform load (ETL) frontends, where the data packet is converted from a platform dependent form into a platform independent form and loaded into a queue. Handlers then process the converted data packets. It is emphasized that this abstract is provided to comply with the rules requiring an abstract that will allow a searcher or other reader to quickly ascertain the subject matter of the technical disclosure. It is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims.

主权项

1. A processing pipeline apparatus, comprising:
computer hardware configured to implement a first load balancer configured to direct a declarative and defined data packet, to one of a plurality of collectors, wherein each collector in the plurality is configured to verify that the data packet is authorized to access a private cloud-based processing pipeline, and wherein the first load balancer and a second load balancer are connected via the plurality of collectors; computer hardware configured to implement the second load balancer configured to receive the data packet from a collector in the plurality and to direct the data packet to one of a plurality of frontends, wherein each of the frontends is configured to convert the data packet from a platform dependent form into a platform independent form and load the resulting converted data packet into a queue; and computer hardware configured to implement a plurality of handlers, wherein one or more handlers in the plurality are configured to process the data packet within the queue.