Elimination of duplicate objects in storage clusters

摘要

Digital objects within a fixed-content storage cluster use a page mapping table and a hash-to-UID table to store a representation of each object. For each object stored within the cluster, a record in the hash-to-UID table stores the object's hash value and its unique identifier (or portions thereof). To detect a duplicate of an object, a portion of its hash value is used as a key into the page mapping table. The page mapping table indicates a node holding a hash-to-UID table indicating currently stored objects in a particular page range. Finding the same hash value but with a different unique identifier in the table indicates that a duplicate of an object exists. Portions of the hash value and unique identifier may be used in the hash-to-UID table. Unneeded duplicate objects are deleted by copying their metadata to a manifest and then redirecting unique identifiers to point at the manifest.

主权项

1. A method of deleting a duplicate of a first digital object within a storage cluster, said method comprising:
receiving a first unique identifier that identifies the location of a first digital object within said storage cluster; receiving a second unique identifier that identifies the location of a second digital object within said storage cluster, wherein said digital objects being duplicates; storing metadata associated with said second digital object in association with metadata associated with said first digital object in a metadata storage location; creating a reference associated with said metadata storage location that identifies said location of said first digital object; deleting said second digital object from said storage cluster; and redirecting said second unique identifier such that said second unique identifier now identifies said metadata storage location, whereby said second unique identifier identifies said first digital object via said metadata storage location and said reference.