INTEGRATING EVENT PROCESSING WITH MAP-REDUCE

摘要

Large quantities of data can be processed and/or queried relatively quickly using a combination of continuous event processing and a Map-Reduce algorithmic tool. The continuous event processor can continuously produce real-time results by merging (a) CQL query results from events received since a currently executing Map-Reduce job was started with (b) a most recent query result produced by a most recently completed Map-Reduce job. When the currently executing Map-Reduce job completes, its query result can be stored and made accessible to the continuous event processor, and a new Map-Reduce job can be started relative to event data that has grown in size since the execution of the last Map-Reduce job. The Map-Reduce algorithmic tool provides a convenient mechanism for analyzing and processing large quantities of data.

主权项

1. A computer-implemented method comprising:
starting a first batch operation that executes a first query relative to event data that has been stored in a persistent data store as of a first time; while the first batch operation is executing, continuously executing a second query relative to event data that is continuously being received from a data stream; while the first batch operation is executing, continuously merging results of the second query with a result of a second batch operation that has completed; continuously outputting results of the merging; and in response to a completion of the first batch operation, replacing the result of the second batch operation with a result of the first batch operation.