Sampling based elimination of duplicate data

摘要

A technique for eliminating duplicate data is provided. Upon receipt of a new data set, one or more anchor points are identified within the data set. A bit-by-bit data comparison is then performed of the region surrounding the anchor point in the received data set with the region surrounding an anchor point stored within a pattern database to identify forward/backward delta values. The duplicate data identified by the anchor point, forward and backward delta values is then replaced in the received data set with a storage indicator.

主权项

1. A method for removing duplicate data stored on a storage system, the method comprising:
performing an operation on a first data set to identify an anchor within the first data set, wherein the anchor defines a starting point in a first region of the first data set for potential data de-duplication; determining a number of consecutive bits or bytes of data that match between the first data set and a second data set forwards and backwards from the identified anchor; and replacing the matching data in the first data set with an indication of the second data set, the anchor, and the number of matching bits or bytes forwards from the anchor and the number of matching bits or bytes backwards from the anchor.