Chained replication techniques for large-scale data streams

摘要

A replication chain comprising one or more replication nodes of a multi-tenant stream management system is assigned to store data records of a partition of a particular data stream. A data record of the partition is received at a selected replication node of the replication chain. In a sequential order, a respective replica of the data record is stored at each replication node of the chain. An acknowledgement of a successful storage of the data record is provided after the replications are completed.

主权项

1. A system, comprising:
one or more computing devices configured to:
assign, to store respective replicas of data records of a partition of a particular data stream, a replication chain comprising three or more replication nodes of a multi-tenant stream management system, wherein the replication chain comprises a head replication node and a tail replication node, and wherein at least one replication node of the plurality of replication nodes is implemented at least in part using a resource shared with a different replication node assigned to a different partition of a different data stream;receive, at an ingestion node of the stream management system assigned to the first partition, a submission request indicating a particular data record of the first partition;receive the data record at the head replication node from the ingestion node;store, in a sequential order starting from the head replication node and ending at the tail replication node, a replica of the data record at a respective local storage device at each of the three or more replication nodes of the replication chain;subsequent to storing the replica of the data record at each of the three or more replication nodes in the sequential order, transmit, from the tail replication node to the ingestion node, an indication that replication of the data record has been completed; andprovide, to a source of the submission request, a response indicating that the data record has been successfully stored at the stream management system.