Stream processing with context data affinity

摘要

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for stream processing with contextual data affinity. One of the methods includes receiving an event at a computing node of a stream processing system that includes one or more computing nodes and data representing a topology of functional modules. At a particular functional module of the topology, an operation that requests access to context data related to the event is determined based on a data attribute of the event. The context data is partitioned among the one or more computing nodes. Context data related to the event is obtained from the computing node. The operation of the particular functional module is processed using the context data, and a modified event is generated. The modified event is provided to a subsequent functional module according to the topology of functional modules.

主权项

1. A method comprising:
receiving an event at a computing node of a stream processing system, wherein the stream processing system comprises one or more computing nodes and data representing a stream processing topology of functional modules, wherein each functional module defines one or more operations to be performed on an event and an output destination for the event, and wherein all functional modules of the stream processing topology are installed on each of the computing nodes; at a particular functional module of the topology, determining that an operation requests access to context data related to the event based on a data attribute of the event, wherein the context data is partitioned among the one or more computing nodes, and wherein each partition of context data is stored in operational memory of the respective computing node; obtaining, from the operational memory of the computing node, the context data related to the event; processing the operation of the particular functional module using the context data obtained from the operational memory of the computing node including generating a modified event; and providing the modified event to a subsequent functional module according to the topology of functional modules.