Parallel upload and download of large files using bittorrent

摘要

Several different embodiments of a segmented object storage system are described. The object storage system divides files into a number of object segments, each segment corresponding to a portion of the object, and stores each segment individually in the cloud storage system. The system also generates and stores a manifest file describing the relationship of the various segments to the original data file. Requests to retrieve the segmented file are fulfilled by consulting the manifest file and using the information from the manifest to reconstitute the original data file from the constituent segments. Modifying, appending to, or truncating the object is accomplished by manipulating individual segments and the manifest file. In further embodiments, manipulation of the individual object segments and/or the manifest is used to implement copy-on-write, snapshotting, software transactional memory, and peer-to-peer transmission of the large file.

主权项

1. A method for storing large object data, comprising:
receiving a request to upload an object from a user device to network-accessible storage over a network; determining the object includes an object size exceeding a predetermined size; dividing the object into a plurality of object segments, wherein each object segment includes a segment size below the predetermined size, wherein the segment size for a respective object segment is equal to or less than a size of a cache of an object storage entity receiving the respective object segment, and wherein each object segment has a file name, the file name including a timestamp identifying a creation time of the respective object segment and a size of the respective object segment; generating a manifest file related to the object, wherein the manifest file allows the plurality of object segments to be retrieved from the network-accessible storage and provided as the object, and wherein the manifest file comprises data verification information for the object and file name information for each object segment; providing the manifest file to a plurality of object storage entities, wherein a total number of object storage entities in the plurality of object storage entities is equal to a total number of object segments; uploading the plurality of object segments to the network-accessible storage over the network, wherein each of the plurality of object storage entities stores a different uploaded object segment in the plurality of object segments, wherein each respective uploaded object segment in the plurality of object segments is uploaded to a respective object storage entity in a first zone, wherein the each respective uploaded object segment in the first zone is subject to a correlated loss of access based on a single event; and replicating the each respective uploaded object segment in the first zone to a respective object storage entity in a second zone, wherein the first zone and the second zone for the each respective uploaded object segment is determined based on a hash operation performed on an identifier associated with the each respective uploaded object segment.