Apparatus, systems, and methods for batch and realtime data processing

摘要

A traditional data processing system is configured to process input data either in batch or in real-time. On one hand, a batch data processing system is limiting because the batch data processing often cannot take into account any data received during the batch data processing. On the other hand, a real-time data processing system is limiting because the real-time system often cannot scale. The real-time data processing system is often limited to dealing with primitive data types and/or a small amount of data. Therefore, it is desirable to address the limitations of the batch data processing system and the real-time data processing system by combining the benefits of the batch data processing system and the real-time data processing system into a single data processing system.

主权项

1. A computing system for generating a summary data of a set of data, the computing system comprising:
one or more processors configured to run one or more modules stored in non-tangible computer readable medium, wherein the one or more modules are operable to:
receive a first set of data and a second set of data, wherein the first set of data comprises a larger number of data items compared to the second set of data;process the first set of data to format the first set of data into a first structured set of data;generate a first summary data using the first structured set of data by operating rules for summarizing the first structured set of data, and store the first summary data in a data store;process the second set of data to format the second set of data into a second structured set of data;generate a second summary data based on the first structured set of data and the second structured set of data by operating rules for summarizing the first structured set of data and the second structured set of data;determine a difference between the first summary data and the second summary data; andupdate the data store based on the difference between the first summary data and the second summary data.