Dynamic Service Fault Detection and Recovery Using Peer Services

摘要

Techniques are described for identifying unhealthy nodes in a multi-node system. One or more parameters of each node is monitored, then compared with the values for the same parameter running on other nodes in the multi-node system. Based on the comparison, a determination is made whether a node is healthy. If the multi-node system comprises one or more nodes with differing capabilities, an adjustment is performed to account for the differing capabilities of each respective node. Further provided are methods of taking remedial action upon a determination that a node is unhealthy. A tuner is used to modify values of health parameters until the node is performing similarly to its peers.

主权项

1. A method of identifying unhealthy nodes in a multi-node system, comprising:
monitoring one or more health parameters of each node of a plurality of nodes in the multi-node system; wherein the plurality of nodes include a first node and one or more other nodes; performing a comparison between the health parameters of the first node and the health parameters of the one or more other nodes; based at least in part on the comparison, determining whether the first node is an unhealthy node; and responsive to determining that the first node is an unhealthy node, performing a remedial action relative to the first node; wherein the method is performed automatically by one or more computing devices.