SCALING EVENT PROCESSING USING DISTRIBUTED FLOWS AND MAP-REDUCE OPERATIONS

摘要

Some event ordering requirements can be determined based on continuous event processing queries. Other event ordering requirements can be determined based on distribution flow types being used to distribute events from event streams to node executing the queries. Events from event streams can be ordered according to ordering semantics that are based on a combination of all of these event ordering requirements. Additionally, virtual computing nodes can be associated with constraints, and computing processors can be associated with capabilities. Virtual computing nodes for processing event streams can be assigned to execute on various computing processors based on both these constraints and capabilities. Additionally, for each of several events in an event stream, a ratio between a total latency and a communication latency can be for determined. Based on an average of these ratios, a quantity of reducing nodes that will be involved in a map-reduce operation can be selected.

主权项

1. A method comprising:
determining a first event ordering requirement based on a first continuous event processing query; determining a second event ordering requirement based on a second continuous event processing query; determining a first type of distribution flow to be used to distribute events from a first event stream to at least a first computing node of a plurality of computing nodes that are configured to execute the first continuous event processing query; determining a second type of distribution flow to be used to distribute events from a second event stream to at least a second computing node of the plurality of computing nodes that are configured to execute the second continuous event processing query; ordering, at the first computing node, events from the first event stream according to a first ordering semantic based at least in part on the first event ordering requirement and the first type of distribution flow, the ordering comprising:
partitioning the events of the first event stream into a first event and at least a second event for processing across the first computing node and at least one additional computing node of the plurality of computing nodes;transmitting the first event for processing by the first computing node; andtransmitting at least the second event for processing by the additional computing node, the first event and the second event processed in accordance with the first ordering semantic; and ordering, at the second computing node, events from the second event stream according to a second ordering semantic based at least in part on the second event ordering requirement and the second type of distribution flow, the second ordering semantic being different from the first ordering semantic.