DISTRIBUTED DATABASE WITH MODULAR BLOCKS AND ASSOCIATED LOG FILES

摘要

A system includes a master node and worker nodes controlled by the master node. Each worker node stores 25 or more modular blocks of a distributed database, where each modular block has a size of 5G bytes or less and has an associated log file. The invention is tailored towards analytics and data warehousing workloads. For example, data analytics workloads usually have data that has a time dimension, and this data is loaded in bulk into the system at regular intervals. An embodiment of the invention leverages these properties to propose new partitioning methods. This invention is designed to scale to hundreds of servers over standard network connections. An embodiment of the invention minimizes the network I/O required to perform large table joins.