Hardware Efficient Fingerprinting

摘要

An approach for fingerprinting large data objects at the wire speed has been disclosed. The techniques include Fresh/Shift pipelining, split Fresh, optimization, online channel sampling, and pipelined selection. The architecture can also be replicated to work in parallel for higher system throughput. Fingerprinting may provide an efficient mechanism for identifying duplication in a data stream, and deduplication based on the identified fingerprints may provide reduced storage costs, reduced network bandwidth consumption, reduced processing time and other benefits. In some embodiments, fingerprinting may be used to ensure or verify data integrity and may facilitate detection of corruption or tampering. An efficient manner of generating fingerprints (either via hardware, software, or a combination) may reduce a computation load and/or time required to generate fingerprints.

主权项

1. A system comprising:
a fingerprint pipeline configured to compute fingerprints for a data chunk; a plurality of sampling modules communicatively coupled with the fingerprint pipeline, the plurality of sampling modules configured to sample candidate fingerprints for generating a sketch for the data chunk; and a fingerprint selection module communicatively coupled with the plurality of sampling modules, the fingerprint selection module configured to select a plurality of fingerprints to create a sketch of the data chunk.