MULTI-LEVEL INLINE DATA DEDUPLICATION

摘要

Technologies are presented for data deduplication that operates at relatively high throughput and with relatively less storage space than conventional techniques. Building upon content-dependent chunking (CDC) using Rabin fingerprints  data may be fingerprinted and stored in variable-size chunks. In some examples  data may be chunked on multiple levels  for example  two levels  variable size large chunks in the first level and fixed-size sub-chunks in the second level  in order to prevent sub-chunks common to two or more data chunks from not being deduplicated. For example  at a first level  a CDC algorithm may be employed to fingerprint and chunk data in content-dependent sizes (variable sizes)  and at a second level the CDC chunks may be sliced into small fixed-size chunks. The sliced CDC chunks may then be used for deduplication.