Adaptive task scheduling of Hadoop in a virtualized environment

摘要

A control module is introduced to communicate with an application workload scheduler of a distributed computing application, such as a Job Tracker node of a Hadoop cluster, and with the virtualized computing environment underlying the application. The control module periodically queries for resource consumption data, such as CPU utilization, and uses the data to calculate how MapReduce task slots should be allocated on each task node of the Hadoop cluster. The control module passes the task slot allocation to the application workload scheduler, which honors the allocation by adjusting task assignments to task nodes accordingly. The task nodes may also activate and deactivate task slots according to the changed slot allocation. As a result, the distributed computing application is able to scale up and down when other workloads sharing the virtualized computing environment change.

主权项

1. A method for managing resources for a first application comprising a distributed computing application, the method comprising:
receiving a first performance metric for a host computer having a first plurality of virtual machines (VMs) controlled by the distributed computing application and a second plurality of VMs controlled by a second application separate from the distributed computing application; determining a state of resource contention between the distributed computing application and the second application based on the first performance metric, wherein it is determined that there is no state of resource contention if VMs controlled by the distributed computing application are primary contributors to the first performance metric; modifying an allocation of task slots associated with the host computer based on the first performance metric, wherein each task slot represents a capability to receive a unit of work for the distributed computing application; and transmitting the modified allocation of task slots to a scheduler, wherein the scheduler is configured to assign a plurality of tasks to be executed in the first plurality of VMs controlled by the distributed computing application based on the modified allocation of task slots.