Data processing apparatus and method of processing data

摘要

Data processing apparatus comprising: a chunk store having a plurality of chunk sections, each operable to store specimen data chunks, the apparatus being operable to: process an input data set into one or more input data chunks; identify a specimen data chunk in one of said chunk sections which corresponds to a first input data chunk; identify a second input data chunk not corresponding to a specimen data chunk in the chunk store; and store the second input data chunk as a specimen data chunk in proximity to the identified specimen data chunk corresponding to the first input data chunk.

主权项

1. An apparatus comprising:
a chunk store having a plurality of chunk sections each storing specimen data chunks; a manifest store for containing a manifest representing at least part of a data set and having references to said chunk sections; at least one processor configured to: process an input data set into input data chunks; identify, using the manifest, a specimen data chunk in a given one of said chunk sections which corresponds to a first of the input data chunks; identify a second of the input data chunks not corresponding to a specimen data chunk in the chunk store; store the second input data chunk as a specimen data chunk in deliberate proximity to the identified specimen data chunk, wherein the storing in deliberate proximity results in selecting the given chunk section rather than another of said chunk sections to store the second input data chunk as a specimen data chunk; associate a specimen data chunk in at least one chunk section with a back-reference to a manifest referencing that specimen data chunk; determine when a given specimen data chunk is not associated with a back-reference to a manifest; delete the given specimen data chunk from a particular chunk section after a predetermined time period or number of iterations in response to determining that the given specimen data chunk is not associated with a back-reference to a manifest; and after the deleting, reduce fragmentation of the particular chunk section by rearranging chunks remaining in the particular chunk section.