Apparatus and method for processing data streams

摘要

A distributed architecture and method for maintaining the integrity of data streams within a multi-pipelined processing environment. The architecture comprising a communications network for carrying a plurality of data streams and a master processor adapted to process one or more messages in at least one of the data streams, the message processing including the creation of one or more data packets within the stream, each packet encapsulating at least a transaction summary of the data that has been processed. The architecture further comprising at least one slave processor per master processor adapted to emulate the transactional state of the master processor by regenerating the data stream as a result of processing the one or more data packets, whereupon in response to an error event on the master processor, the slave processor acts to avoid interrupting the data stream by generating one or more successive data packet(s). Hence, the architecture and method serve as a high availability, robust fault tolerant system, mitigating against the loss of data within data streams.

主权项

1. A distributed architecture configured to maintain the integrity of data streams within a multi-pipelined processing environment, the architecture comprising:
a communications network for carrying a plurality of data streams; a master processor adapted to process one or more messages in at least one of the data streams, the message processing including the creation of one or more data packets within the at least one data stream, each packet encapsulating a transaction unit for the message processing, the transaction unit including the message that triggered the message processing, a description of changes or updates to data made as a result of the message processing and a message configured to trigger subsequent processing; and at least one slave processor adapted to emulate the transactional state of the master processor by regenerating the at least one data stream as a result of processing the one or more data packets by storing the description of changes or updates to the data made as a result of the message processing and storing the message configured to trigger subsequent processing in a processing queue, whereupon in response to an error event on the master processor, the slave processor acts to avoid interrupting the at least one data stream by generating one or more successive data packet(s) using only data previously received from the master processor.