Self healing and restartable multi-steam data backup

摘要

Implementations are provided herein for a self-healing scalable ring communication topology that enables a multi-stream restartable backup. A standard network management protocol tree-walk process can be altered to support parallel tree-walk. Parallel tree-walk provides for splitting a backup directory among multiple sessions, where each session, in parallel, can walk the tree of portions of a backup directory and stream the results to separate backup storage devices. By allowing multiple backup sessions to work together to back up a single root directory, the backup process becomes more scalable to very large scale data storage systems. In addition, if a single stream of a multi-stream backup experiences a failure, only that stream need be restarted and other streams of the backup can continue without or with very little interruption.

主权项

1. A method comprising:
receiving a multi-stream backup request associated with a dataset; establishing a first session and a second session among a set of sessions; associating a first stream identifier with the first session and a second stream identifier with the second session; dynamically assigning sessions among the set of sessions with an upstream neighbor and a downstream neighbor based at least in part on associated stream identifiers, wherein each session in the set of sessions has one upstream neighbor and one downstream neighbor; splitting the dataset into a first segment and a second segment among a set of segments based on sequential hash values, wherein segments in the set of segments are associated with at least a segment identifier, a starting hash value, and an ending hash value; storing at least the segment identifier, the starting hash value, and the ending hash value for segments in the set of segments in a segment location table; assigning the first segment to the first session and the second segment to the second session; in parallel, streaming the first segment by the first session to a first backup storage drive among a set of backup storage drives and the second segment by the second session to a second backup storage drive among the set of backup storage drives, wherein the streaming is based on tree-walking a segment, and wherein the tree-walking the segment is based on sequential hash values of the segment; and in response to streaming an entirety of a segment among the set of segments by a session among the set of sessions:
communicating among sessions in the set of sessions that the segment cannot be split; andrequesting a new segment by the session from at least one of the upstream neighbor of the session or the downstream neighbor of the session.