Data storage system and method by shredding and deshredding

摘要

A system and method for data storage by shredding and deshredding of the data allows for various combinations of processing of the data to provide various resultant storage of the data. Data storage and retrieval functions include various combinations of data redundancy generation, data compression and decompression, data encryption and decryption, and data integrity by signature generation and verification. Data shredding is performed by shredders and data deshredding is performed by deshredders that have some implementations that allocate processing internally in the shredder and deshredder either in parallel to multiple processors or sequentially to a single processor. Other implementations use multiple processing through multi-level shredders and deshredders. Redundancy generation includes implementations using non-systematic encoding, systematic encoding, or a hybrid combination. Shredder based tag generators and deshredder based tag readers are used in some implementations to allow the deshredders to adapt to various versions of the shredders.

主权项

1. A system for use with data, the system comprising a storage unit, at least one processor coupled to the storage unit, and a memory comprising instructions executable by the at least one processor, when executed by the at least one processor, the instructions performing a shredding process and a deshredding process,
the shredding process comprising: (a) a plurality of data operations performable on the data in accordance with a shred order of operations to produce a plurality of shreds, the plurality of data operations comprising at least one transformation operation, and at least one redundancy operation; (b) a tag generation process configured to generate a plurality of tags for the plurality of shreds, one or more tags being associated with each of the plurality of shreds, the tag generation process generating at least one tag after the data operations are performed on the data in accordance with the shred order of operations, the at least one tag comprising information identifying the data operation performed, the shred order of operations being determinable from the one or more tags associated with each of the plurality of shreds; and (c) a storage operation configured to store the plurality of shreds and the plurality of tags in the storage unit, the deshredding process comprising: (a) a retrieval process configured to retrieve a portion of the plurality of shreds and the one or more tags associated with each of the portion of the plurality of shreds from the storage unit; (b) a plurality of inverse data operations comprising for each of the plurality of data operations of the shredding process, an operation that is an inverse of the data operation; and (c) a tag reading process, for each of a portion of the plurality of shreds, the tag reading process being configured to: read the one or more tags associated with the shred, identify the plurality of data operations performed to produce the shred, identify ones of the plurality of inverse data operations that are inverses of the plurality of data operations performed to produce the shred, and execute each of the ones of the plurality of inverse data operations identified as being inverses of the plurality of data operations performed to produce the shred in accordance with a deshred order of operations to recover the data from the portion of the plurality of shreds, the deshred order of operations being an inverse of the shred order of operations.