System and method for logstream archival

摘要

A method of archiving data from a logstream includes determining a begin time parameter earlier in time than a last record obtained from the logstream of data. A request is sent for records from the logstream of data starting at the begin time parameter. Records are received from the logstream of data and selected records are archived.

主权项

1. A machine implemented method of archiving data into a storage device, the method comprising:
obtaining a first timestamp corresponding to a last record archived from a logstream of data, the first timestamp indicating when the last record was received at the logstream and is recorded as a bookmark in the storage device, the logstream of data including records stored in order according to when a record of the records was received at the logstream, wherein the logstream of data comprises a continuous stream of data without discrete demarcation of a start and an end of the data, and wherein each record includes a timestamp indicating when the record was received at the logstream; modifying the first timestamp to an earlier point in time; requesting records from the logstream that include a second timestamp indicating a time at the earlier point in time and requesting records from the logstream that include a third timestamp indicating a time after the earlier point in time and before the first timestamp; receiving the requested records from the logstream; determining a first record of the requested records received that has not previously been archived, wherein the first record includes the timestamp indicating when the first record was received at the logstream; archiving the first record not previously archived; archiving the requested records received that include a timestamp that is the same as the timestamp indicating when the first was received at the logstream and the requested records received that include a timestamp that indicates a time after the timestamp indicating when the first record was received at the logstream; and recording the third timestamp with the bookmark.