EXACTLY-ONCE SEMANTICS FOR STREAMING ANALYTICS IN NON-IDEMPOTENT OUTPUT OPERATIONS

摘要

An approach for distributed stream computing in non-idempotent output operations is provided. The approach assigns an eventid to a corresponding entityid. The approach determines a minibatchid and a partitionid for a partition. The approach determines whether the partition was previously processed. The approach generates a new minibatchid and a new partitionid for a new partition based upon determining the partition was not previously processed. The approach determines whether a record was previously processed based upon determining the partition was previously processed. The approach processes the record of the partition based upon determining the record was not previously processed.

主权项

1. A distributed stream computing method in non-idempotent output operations, implemented by one or more processors, the method comprising:
assigning an eventid to a corresponding entityid; determining a minibatchid and a partitionid for a partition; determining whether the partition was previously processed; generating a new minibatchid and a new partitionid for a new partition based upon determining the partition was not previously processed; determining whether a record was previously processed based upon determining the partition was previously processed; and processing the record of the partition based upon determining the record was not previously processed.