MANAGING SEQUENTIAL DATA STORE

摘要

The disclosure is directed to storing a data log file (“log”) in a distributed computing system. The log can be stored as a sequence of data records (“records”). A number of writer computers (“writers”) can write to a log, and a number of reader computers (“readers”) can read from the log. A log sequencer computer (“log sequencer”) facilitates storage of the records in data storage nodes of the distributed computing system. When the log sequencer receives a request for appending a record to a log, the log sequencer assigns a sequence number to the record that is unique and increases monotonically within the log. When a reader requests the log, the records can be delivered to the reader in the order of their sequence numbers. Further, since the records are associated with sequence numbers, a reader can seek to arbitrary sequence number within the log.

主权项

1. A method performed by a computing system, comprising:
receiving multiple data records associated with a data log file at a log sequencer computer in a distributed computing system, the data log file to be stored as a sequence of the data records; assigning, by the log sequencer computer, a sequence number to a first data record of the data records, wherein the sequence number includes an epoch number and a count, the epoch number indicative of a series of the data records of the data log file, the count indicative of a sequence of the first data record in the series, the count being increased monotonically within the series at the arrival of every data record at the log sequencer computer; determining, by the log sequencer computer and using a non-deterministic method, a set of multiple data storage nodes at which the data records is to be stored; and storing the data records at the set of the data storage nodes.