Efficiently storing data in a dispersed storage network

摘要

A method begins by a dispersed storage (DS) processing module ingesting a plurality of data blocks of a large amount of data. The method continues with the DS processing module temporarily storing the plurality of data blocks and determining whether to not permanently store the plurality of data blocks. When the plurality of data blocks is to be permanently stored, the method continues with the DS processing module storing the plurality of data blocks and generating a group of partial redundancy data. When the plurality of data blocks is not to be permanently stored, the method continues with the DS processing module creating a new plurality of data blocks from other data blocks that are to be permanently stored and generating the group of partial redundancy data based on the new plurality of data blocks.

主权项

1. A method for execution by a dispersed storage (DS) unit of a dispersed storage network (DSN), the method comprises:
ingesting a respective section of data, wherein a large amount of data is divided into sets of partitions, wherein each of the sets of partitions is divided into a set of sections, wherein a set of DS units ingests the set of sections, and wherein the set of DS units includes the DS unit; for the ingested respective section of data:
dividing the ingested respective section of data into a plurality of data segments;for a data segment of the plurality of data segments:
dividing the data segment into a plurality of data blocks;temporarily storing the plurality of data blocks;determining whether to not permanently store one or more of the plurality of data blocks;when the one or more of the plurality of data blocks is to be permanently stored:
storing the one or more of the plurality of data blocks; andgenerating a group of partial redundancy data based on the one or more of the plurality of data blocks and in accordance with a dispersed storage error coding function; andwhen the one or more of the plurality of data blocks is not to be permanently stored:
creating a new plurality of data blocks from data blocks of the plurality of data blocks that are to be permanently stored and data blocks from another data segment that are to be permanently stored; andgenerating the group of partial redundancy data based on the new plurality of data blocks in accordance with the dispersed storage error coding function.