| 摘要 |
A sampling based technique for eliminating duplicate data (de-duplication) stored on storage resources, is provided. According to the invention, when a new data set, e.g., a backup data stream, is received by a server, e.g., a storage system or virtual tape library (VTL) system implementing the invention, one or more anchors are identified within the new data set. The anchors are identified using a novel anchor detection circuitry in accordance with an illustrative embodiment of the present invention. Upon receipt of the new data set by, for example, a network adapter of a VTL system, the data set is transferred using direct memory access (DMA) operations to a memory associated with an anchor detection hardware card that is operatively interconnected with the storage system. The anchor detection hardware card may be implemented as, for example, a FPGA is to quickly identify anchors within the data set. As the anchor detection process is performed using a hardware assist, the load on a main processor of the system is reduced, thereby enabling line speed de-duplication. |
| 主权项 |
1. A method, comprising:
receiving a data set at a storage system; in response to receiving the data set, detecting one or more anchor locations in the data set by an anchor detection circuitry of the storage system, wherein the anchor detection circuitry is a hardware device dedicated to detecting the one or more anchor locations; transferring each anchor locations to a data de-duplication module executed by a processor of the storage system; determining if the anchor location is located in a database; if the anchor location is not located in the database, adding the anchor location to the database; if the anchor location is located in the database, examining, by the de-duplication module executed by the processor, bits preceding the anchor location utilizing a first delta value and the bits following the anchor location utilizing a second delta value to identify duplicate data; and in response to identifying the duplicate data, removing the duplicate data from the data set prior to storing the data set on a storage device operatively connected to the storage system.
|
