Computer system and process for transferring multiple high bandwidth streams of data between multiple storage units and multiple applications in a scalable and reliable manner

摘要

Multiple applications request data from multiple storage units over a computer network. The data is divided into segments and each segment is distributed randomly on one of several storage units, independent of the storage units on which other segments of the media data are stored. At least one additional copy of each segment also is distributed randomly over the storage units, such that each segment is stored on at least two storage units. This random distribution of multiple copies of segments of data improves both scalability and reliability. When an application requests a selected segment of data, the request is processed by the storage unit with the shortest queue of requests. Random fluctuations in the load applied by multiple applications on multiple storage units are balanced nearly equally over all of the storage units. This combination of techniques results in a system which can transfer multiple, independent high-bandwidth streams of data in a scalable manner in both directions between multiple applications and multiple storage units.

主权项

1. A data storage system, comprising:
a plurality of client systems, each client system having a file system through which applications executed on the client system access data; a plurality of storage servers coupled to the plurality of client systems via a computer network, each storage server storing data accessible through the file system; wherein the stored data is divided into segments that are stored across the plurality of storage servers, with two or more copies of each segment being distributed among the plurality of storage servers, such that each segment is stored on at least two of the storage servers and wherein the segments of data are distributed nonsequentially among the plurality of storage servers; and storage configured to store information indicating the storage servers on which the segments of the stored data are stored; at least one of the client systems being configured to:
access, before reading data, the stored information indicating the storage servers on which the segments of the data are stored, andcommunicate directly with the storage servers to request the segments of the data using the accessed information; and each storage server being configured to:
maintain data defining information that associates, for each segment stored on the storage server, the identifier of the segment with the location of the segment in storage;in response to a request from one of the client systems for a segment, wherein the received request includes the identifier of the requested segment, determine the location of the segment in the storage using the information that associates the identifier of the requested segment with the location of the segment in the storage;retrieve the requested segment from the storage; andsend the retrieved segment over the computer network to the client system that requested the segment.