EFFICIENT DATA DEDUPLICATION

摘要

Efficient data deduplication is described herein. A deduplication bit array partition can be created that corresponds to a number of data items in an expected dataset. The deduplication bit array partition can track whether the data items have been received. When a data item in the expected dataset is received, a bit in the deduplication bit array partition corresponding to the received data item can be accessed to determine, based on the value of the bit, if the received data item has already been received. When the value of the bit indicates that the received data item has not already been received, the value can be changed to indicate that the data item has now been received. When the value of the bit indicates that the received data item has already been received, the data item can be deleted or ignored.

主权项

1. A computer-implemented method for deduplicating a dataset, the method comprising:
receiving a number of data items in an expected dataset; creating a deduplication bit array partition corresponding to the number of data items in the expected dataset, the deduplication bit array partition tracking whether the data items have been received; receiving a data item in the expected dataset; accessing the deduplication bit array partition to determine if the received data item has already been received; and upon determining that the received data item has not already been received, changing a value of a bit in the deduplication bit array partition corresponding to the received data item.