CONTAINER VIRTUAL MACHINES FOR HADOOP

摘要

A distributed computing application is described that provides a highly elastic and multi-tenant platform for Hadoop applications and other workloads running in a virtualized environment. Data and compute nodes are separated into different virtual machines (VM). Compute VMs are used to launch containers from different tenants. Compute VMs are organized in pools of hot spare VMs that are immediately available for launching a container and executing a task, and pools of cold spare VMs. Each compute VM may include a mounted network filesystem provided by a node manager to share intermediate outputs across VMs executing on the same host.

主权项

1. A method for executing a job comprising a plurality of tasks executed in parallel on a distributed computing application, the method comprising:
generating a list of available virtual machines (VMs) in a powered on state and executing on a first host; responsive to receiving a request to execute a first task of the plurality of tasks associated with a first tenant on the first host, allocating a first VM from the list of available VMs; modifying the first VM to mount a network filesystem shared by VMs executing on the first host and associated with the first tenant; and launching, in the first VM, a container process that executes the first task of the plurality of tasks associated with the first tenant.