DATA DEDUPLICATION IN A FILE SYSTEM

摘要

A data deduplication capability is presented. The data deduplication capability enables deduplication of data of a set of files, where the set of files may include files stored in network-based data storage elements and, optionally, files stored in one or more client devices which may communicate with the network-based data storage elements. The data deduplication capability may use one or more data deduplication techniques within files (for intra-file redundancy) or across files (for inter-file redundancy) in order to reduce or even minimize storage cost associated with storage of the files or bandwidth cost associated with transfers of the files. The data deduplication capability may use one or more data deduplication techniques in conjunction with one or more data compression techniques in order to reduce or even minimize storage cost associated with storage of the files or bandwidth cost associated with transfers of the files.

主权项

1. An apparatus, comprising:
a processor and a memory communicatively connected to the processor, the processor configured to:
receive a file comprising original file contents;determine a set of data chunks of the original file contents of the file and a respective set of hash values of the data chunks;determine whether the data chunks are stored in a data chunk store comprising a set of data chunks for one or more stored files;encode the original file contents of the file, to form an encoded form of the original file contents of the file, based on the hash values of the data chunks;compress the encoded form of the original file contents of the file to form a compressed and encoded form of the original file contents of the file; andstore the compressed and encoded form of the original file contents of the file.