Data duplication in tape drives

摘要

A method and apparatus for data de-duplication is disclosed. Use de-duplication engine (DDE) divides data into regions. The DDE processes the regions in a serial fashion. The DDE clears its hash table before processing the first region. Each region is divided into a number of chunks. A hash is generated for each chunk in a region. When a hash for a chunk is not in a hash table, the hash is stored in the hash table and the chunk is stored on media. When the hash is a duplicate of a hash already in the table, the hash and chunk are discarded and a reference to the previously stored chunk is stored to the media. The DDE does not retain all the hashes from all the regions in the hash table at the same time. The DDE only retains the hashes for the current region being processed and N previous regions where N is a positive integer greater than zero.

主权项

1. A tape drive, comprising:
a de-duplication engine (DDE) to divide data into a plurality of regions; the DDE having a hash table, the DDE clearing the hash table before processing a first of the plurality of regions; the DDE serially processing each of the plurality of regions by:
dividing each region into a plurality of chunks;generating a hash for each of the plurality of chunks in the region;storing each hash that is not currently in the hash table into the hash table and storing the corresponding chunk onto a tape;discarding each hash already stored in the hash table and discarding its corresponding chunk and storing a reference to the previously stored chunk onto the tape: the DDE retaining in the hash table only the hashes for a current region being processed and N previous regions where N is a positive integer greater than zero.