Efficient De-Duping Using Deep Packet Inspection

摘要

The efficiency of data de-duplication may be improved by storing related file data in a single container, or in multiple linked containers, of a history. Additionally, the efficiency of data de-duplication may be improved when shorter hash tables are used to reference historical data in a history. Shorter hash tables may be achieved by storing fewer than all the hash values obtained for a given amount of historical data. Further, the efficiency of data de-duplication may be improved by comparing related incoming file data with historical data from a container without hashing/chunking the remaining file data upon matching an earlier chunk of the incoming file data to the container.

主权项

1. A method for performing data de-duplication using deep packet inspection, the method comprising:
receiving file data carried by a data stream; partitioning a portion of file data into a data chunk; applying a hash algorithm to blocks within the data chunk to produce hash values; selecting fewer than all of the hash values to store in a hash table; storing the portion of file data in a container in a history; and storing the selected hash values in an entry of the hash table without storing non-selected hash values in the hash table.