HIGHER EFFICIENCY STORAGE REPLICATION USING COMPRESSION

摘要

In one embodiment, there is a multi-cluster synchronization system between two or more clusters. The multi-cluster synchronization system uses variable compression to optimize the transfer of information between the clusters. Compression is used not only to minimize the total number of bytes sent between the two clusters, but to dynamically vary the size of the objects sent across the wire to optimize for higher throughput after considering packet loss, TCP windows, and block sizes. This includes both the packaging of multiple small files together into one larger compressed file, saving on TCP and header overhead, but also the chunking of large files into multiple smaller files that are less likely to have difficulties due to intermittent network congestion or errors. A further embodiment uses forward error correction to maximize the chances that the remote end will be able to correctly reconstitute the transmission.

主权项

1. A multi-cluster synchronization system, comprising:
a first cluster including a first cluster-internal network, the first cluster further including a first structured information repository and a first object storage, wherein the first structured information repository contains metadata corresponding to stored information objects in the first object storage, and wherein the first structured information repository and the first object storage are coupled via the first cluster-internal network; an intercluster network coupling the first cluster and a remote cluster; a network evaluator that determines a state of one or more networks coupled to the first cluster and the remote cluster; and an intercluster repository synchronizer including a compression module that identifies one or more files to compress from the first object storage and transmit to the remote cluster in compressed form, wherein the compression module dynamically determines, based on the state of the one or more networks, a size to compress the one or more identified files.