HANDLING OUT OF ORDER EVENTS

摘要

Processing streaming data in accordance with policies that group data by source, enforce a maximum permissible late arrival value for streaming data, a maximum permissible early arrival for data and/or a maximum degree to which data can be out of order and still be compliant with the out of order policy is described. The correct starting point for reading a data stream so as to produce correct output from a given output start time can be enabled using the early arrival policy. Using combinations of policies, output can be generated promptly (with low latency). When input from a given source is not disrupted, output can be generated with low latency. Output can be generated even when the input stops by applying a late arrival policy.

主权项

1. A system comprising:
at least one processor: a memory connected to the at least one processor; and at least one program module loaded into the memory causing the at least one processor to: ingest data from at least one data stream, the data comprising a plurality of units of data, each unit of data of the plurality of units of data associated with a first time comprising an associated time and a second time comprising an arrival time, the arrival time comprising a time at which a unit of data of the plurality of units of data arrives at a computing device; and evaluate the unit of data with respect to compliance with at least a first policy and a second policy of a plurality of policies.