Map-Reduce Ready Distributed File System

摘要

A map-reduce compatible distributed file system that consists of successive component layers that each provide the basis on which the next layer is built provides transactional read-write-update semantics with file chunk replication and huge file-create rates. A primitive storage layer (storage pools) knits together raw block stores and provides a storage mechanism for containers and transaction logs. Storage pools are manipulated by individual file servers. Containers provide the fundamental basis for data replication, relocation, and transactional updates. A container location database allows containers to be found among all file servers, as well as defining precedence among replicas of containers to organize transactional updates of container contents. Volumes facilitate control of data placement, creation of snapshots and mirrors, and retention of a variety of control and policy information. Key-value stores relate keys to data for such purposes as directories, container location maps, and offset maps in compressed files.

主权项

1. A distributed file system, comprising:
a plurality of containers in which each container stores file and directory meta-data as well as file content data; wherein references to file content data are stored on a subset of nodes on which container meta-data and data are stored; a container location database (CLDB) configured to maintain information about where each of said plurality of containers is located; a plurality of cluster nodes, each cluster node containing one or more storage pools, each storage pool containing zero or more containers; and a plurality of inodes for structuring data within said containers; wherein said CLDB is configured to assign nodes as replicas of data in a container to meet policy constraints.