System and Method For Correlating Cloud-Based Big Data in Real-Time For Intelligent Analytics and Multiple End Uses

摘要

A processing platform integrates ETL (extract, transform, and load), real time stream processing, and “big data” data stores into a high performance analytic system that runs in a public or private cloud. The platform performs real time pre-storage enrichment of data records to form a single comprehensive record usable for analytics, searching and alerting. The platform further supports sharing of components and plug-ins and performs automatic scaling of resources based on real time resource monitoring and analysis.

主权项

1. A process for data collection and conditioning for use in one or more user applications, comprising:
receiving multiple data records from multiple data sources at a processing engine via multiple transport mechanisms, wherein at least some of the multiple data records have different formats from each other and from a pre-established internal data format; parsing in near real time by at least one of multiple parsers each of the multiple data records into multiple constituent parts, wherein each of the multiple parsers is assigned to a different transport mechanism for transporting data records having different formats; translating in near real time by at least one translator each of the multiple data records using their parsed multiple constituent parts into a pre-established internal data format; loading a selected enrichment cache in an enrichment node of the processing engine, including dimension records, and comparing in near real time by the processing engine each of the multiple translated internal data records with the dimension records in the selected enrichment cache to determine applicability to one or more data elements therein; if applicable, enriching in near real time by the enrichment node of the processing engine the one or more data elements in the multiple translated internal data records with additional data pursuant to the dimension records to form one or more enriched translated internal data records; transmitting in near real time the one or more enriched translated internal data records to at least one data sink for storage therein; and accessing in near real time the one or more enriched translated internal data records by one or more applications for use thereby.