Systems, methods, and devices for dynamic resource monitoring and allocation in a cluster system

摘要

In an embodiment, the systems, methods, and devices disclosed herein comprise a computer resource monitoring and allocation system. In an embodiment, the resource monitoring and allocation system can be configured to allocate computer resources that are available on various nodes of a cluster to specific jobs and/or sub-jobs and/or tasks and/or processes.

主权项

1. A computer cluster comprising:
a management computing device comprising a supervisor controller configured to coordinate processing of a plurality of sub-jobs for a plurality of overall jobs; a plurality of computer system nodes configured to communicate with the management computing device, and to perform processing of received sub-jobs, the computing system nodes each comprising:
one or more processors configured to perform computing processes on received sub-jobs;an agent controller comprising:
a monitoring interface configured to monitor utilization by sub-jobs of system resources of a first computing system node; anda reporting controller configured to transmit the monitored system resources utilization to the supervisor controller in substantially real-time; wherein the supervisor controller is configured to assign an additional sub-job to the first computing system node based on determining that the utilization of at least one system resource of the first computing system node is below a threshold level, the determining based on the monitored system resources utilization transmitted from the reporting controller to the supervisor controller; wherein the at least one system resource of the first computing system node is a first electronic random access memory capacity, wherein the supervisor controller is configured to monitor a second electronic random access memory capacity of a second computing system node, wherein the assigning by the supervisor controller of the additional sub-job comprises assigning the additional sub-job to the first computing system node based on determining that utilization of the first electronic random access memory capacity is below the threshold level, wherein the supervisor controller is configured to prevent assignment of additional sub-jobs to the second computing system node based on determining that utilization of the second electronic random access memory capacity is at or above a threshold value, wherein the additional sub-job requires utilization of the first electronic random access memory capacity that is unused on the first computing system node.