Data processing apparatus and method of processing data

摘要

One embodiment is a data processing apparatus that has a chunk store containing specimen data chunks, a manifest store containing a plurality of manifests, each of which represents at least a part of previously processed data and includes at least one reference to at least one of the specimen data chunks, and a sparse chunk index containing information on only some specimen data chunks. Input data is processed into a plurality of input data segments. Each manifest of the first set has at least one reference to one of said specimen data chunks that corresponds to one of the input data chunks of a first input data segment. Specimen data chunks corresponding to other input data chunks of the first input data segment are identified by using the identified first set of manifests and at least one manifest identified when processing previous data.

主权项

1. A data processing apparatus comprising:
a chunk store configured for containing specimen data chunks, a manifest store configured for containing a plurality of manifests, each of which represents at least a part of previously processed data and comprises at least one reference to at least one of said specimen data chunks, a sparse chunk index containing information on only a subset less than all of the specimen data chunks in the chunk store, wherein information contained in the sparse chunk index for a given one of the specimen data chunks includes references to manifests that include a reference to the given specimen data chunk, at least one processor to:
process input data into a plurality of input data segments, each composed of input data chunks;identify a first set of manifests, where each manifest of the first set has at least one reference to one of said specimen data chunks that corresponds to one of the input data chunks of a first input data segment, and on which there is information contained in the sparse chunk index; andidentify specimen data chunks corresponding to other input data chunks of the first input data segment by using the identified first set of manifests and at least one manifest identified when processing previous data.