B-file abstraction for efficiently archiving self-expiring data

摘要

Systems and methods are provided for data processing and storage management. In an illustrative implementation an exemplary computing environment comprises at least one data store, a data processing and storage management engine (B-File engine) and at least one instruction set to instruct the B-File engine to process and/or store data according to a selected data processing and storage management paradigm. In an illustrative operation, the illustrative B-File engine can generate a B-File comprising multiple buckets and store sample items in a random bucket according to a selected distribution. When the size of the B-FILE grows to reach a selected threshold (e.g., maximum available space), the B-File engine can shrink the B-File by discarding the largest bucket. Additionally, the B-File engine can append data to existing buckets and explicitly cluster data when erasing data such that data can be deleted together into the same flash block.

主权项

1. A method comprising:
receiving data items for storage on a solid state storage device having multiple storage blocks; determining different levels for the data items, wherein the different levels determine when the data items expire and are deleted from the solid state storage device; assigning the data items to different storage buckets of a bucket file based on the levels; appending the data items within the different storage buckets on the solid state storage device such that individual storage buckets of data items are clustered together in corresponding blocks of the solid state storage device while maintaining corresponding numbers of the data items in the different storage buckets so that the corresponding numbers of the data items in the different storage buckets progressively decrease by level; and periodically deleting largest storage buckets from the solid state storage device via one or more block erase operations while at least some other storage buckets remain stored on the solid state storage device, the largest storage buckets having greater numbers of the data items than the other storage buckets that remain stored on the solid state storage device.