Dynamic Partitioning of Streaming Data

摘要

A real time data analysis and data filtering system for managing streaming data is presented. The method breaks a stream of data into a set of queues that are themselves streaming data but that are handled separately by unique processing steps. The queues are dynamically created on an as needed basis based on inspection of the data. In this manner the speed and efficiency of parallel processing is applied to serially streaming data. A method of filtering the data to present the new streaming data queues is also described. The method makes use of keys that are used to filter the data stream into individual queues. In another embodiment a pre-processing step includes creation of keys and insertion of keys into the streaming data to enable subsequent data mining to be accomplished on a less powerful computing device.

主权项

1. A method for managing a data stream said method comprising:
a) programming a computing device having a memory, to receive an input data stream from a data source said data stream comprising data items presented serially over time and said data items including data elements, b) programming the computing device to store at least one selected key in the memory of the computing device where the selected key is a possible value for a data element, c) programming the computing device to filter the data stream into separate queues based upon the occurrence of the at least one selected key within a data item, wherein the queues are themselves data streams, d) storing at least one pre-selected action to be performed by the computing device on an at least one pre-selected separated queue, e) performing the pre-selected action on the at least one pre-selected separated queue.