OBJECT-LEVEL IDENTIFICATION OF DUPLICATE DATA IN A STORAGE SYSTEM

摘要

The technique introduced here includes a system and method for identification of duplicate data directly at a data-object level. The technique illustratively utilizes a hierarchical tree of fingerprints for each data object to compare data objects and identify duplicate data blocks referenced by the data objects. The hierarchical fingerprint trees are constructed in such a manner that a top-level fingerprint (or object-level fingerprint) of the hierarchical tree is representative of all data blocks referenced by a storage system. In embodiments, inline techniques are utilized to generate hierarchical fingerprints for new data objects as they are created, and an object-level fingerprint of the new data object is compared against preexisting object-level fingerprints in the storage system to identify exact or close matches. While exact matches result in complete deduplication of data blocks referenced by the data object, hierarchical comparison methods are used for identifying and mapping duplicate data blocks referenced by closely related data objects.