Hierarchical, parallel models for extracting in real-time high-value information from data streams and system and method for creation of same

摘要

A computer system identifies high-value information in data streams. The computer system receives a plurality of filter graph definitions. Each of the filter graph definitions includes a plurality of filter nodes arranged in a two-dimensional graph by a plurality of graph edges. Each filter node is configured to accept or reject individual packets in a data stream based on content and/or metadata information associated with the individual packet. The system prepares the filter graph for execution. In response to receiving a data stream with a plurality of packets, the system distributes the packets to inputs of each of the executable filter graph definitions. The system identifies, using the executable filter graph definitions, packets with high-value information based on parallel execution of the filter nodes included in the respective filter graph definition.

主权项

1. A method for identifying high-value information in data streams, comprising:
at a computer system including a plurality of processors and memory storing programs for execution by the processors:
receiving a plurality of filter graph definitions, wherein each filter graph definition includes a plurality of filter nodes arranged in a two-dimensional graph defined by a plurality of graph edges, wherein the filter nodes include textual filters that reject or accept an individual packet based on text content of the individual packet and at least one of author filters or publisher filters that reject or accept an individual packet based on author or publisher information associated with the individual packet, respectively;performing a continuous monitoring process for a data stream that includes a plurality of packets, including:
without user intervention, in response to receiving the data stream with the plurality of packets, distributing the plurality of packets to inputs of the plurality of executable filter graph definitions; andidentifying, using a respective executable filter graph definition, respective ones of the plurality of packets with high-value information according to the respective executable filter graph definition, based on parallel execution of the filter nodes included in the respective executable filter graph definition, the identifying including:
executing the textual filters on the text content of the plurality of packets; andafter executing the textual filters, executing the author and/or publisher filters on the packets not rejected by the textual filters.