Methods and Systems For Vectored Data De-Duplication

摘要

The present invention is directed toward methods and systems for data de-duplication. More particularly, in various embodiments, the present invention provides systems and methods for data de-duplication that may utilize a vectoring method for data de-duplication wherein a stream of data is divided into “data sets” or blocks. For each block, a code, such as a hash or cyclic redundancy code may be calculated and stored. The first block of the set may be written normally and its address and hash can be stored and noted. Subsequent block hashes may be compared with previously written block hashes.

主权项

1. A computerized method for performing vectored de-duplication, comprising:
comparing an outer de-duplication code for a first entity of data received as part of an input stream to an outer de-duplication code for a previously processed entity of data, where an entity of data comprises two or more consecutive blocks of data; upon determining that the outer de-duplication code for the first entity of data matches the outer de-duplication code for the previously processed entity of data, storing in an output stream an entity vector instead of the first entity of data and removing the first entity of data from the input stream, where the entity vector points in the output stream to one of, the previously processed entity of data, or another entity vector from which the previously processed entity of data can be referenced, where the entity vector is placed in a location in the output data stream where the first entity of data would have been placed, where the entity vector contains fewer bits than the first entity of data, where the input stream can be recreated from the output stream without reference to other de-duplication data structures, and where the output stream includes self-describing data.