| 摘要 |
Provided is a duplicate data deletion method and device. The method includes: partitioning a file to be stored, and calculating a fingerprint of each partition in the partitioning processing result; sampling the fingerprint of each partition, and generating a fingerprint sampling table for the file to be stored according to the sampled fingerprint; determining a similar grouping of the file to be stored in a grouping sampling library according to the fingerprint sampling table and the grouping sampling library; and performing duplicate data deletion on the file to be stored according to the fingerprint data in a fingerprint grouping corresponding to the similar grouping in a fingerprint library. The device includes: a partitioning module, a sampling module, a grouping module and a duplicate data deletion module. The present invention solves the problem in the prior art that the calculation amount and the resource consumption introduced by massive partitioned data during duplicate deletion are huge and reduces the calculation amount of de-duplication during duplicate data deletion. |
