TABLE FORMAT FOR MAP REDUCE SYSTEM

摘要

A key-value store provides column-oriented access to data in a distributed and fault tolerant manner. Data can be inserted into the data store and data can be retrieved either randomly or sequentially from the data store at high rates. Keys for a table are ordered and the entire table is divided into key ranges. Each key range is handled by a table which itself is divided into key ranges called a partition. Partitions are also divided into segments. Such recursive division into smaller and smaller key ranges provides parallelism. At the highest level, operations on tablets can be distributed to different nodes. At lower levels, different threads can handle operations on individual segments. Large-scale restructuring operations can be decomposed into operations on individual segments so that a global lock on larger objects does not need to be kept across the entire operation.

主权项

1. A computer implemented method for column-oriented access to data in a distributed and fault tolerant manner, comprising:
providing a data store into which data is inserted, and from which data is retrieved, randomly or sequentially, at high rates by:
ordering keys in a key-value store for at least one table and dividing the entirety of said table into key ranges;handling each key range by a key range table, wherein said key range table is divided into key ranges comprising at least one partition;at a highest level, distributing operations on tablets to different nodes;at lower levels, using different threads to handle operations on individual partitions or portions of partitions;decomposing large-scale restructuring operations into operations on individual partitions or portions of partitions; andintegrating said key-value store tightly into said data store's file system; wherein standard file system operations are performed directly on said key-value store tables.