Distributed computing fault management

摘要

An automated system may be employed to perform detection, analysis and recovery from faults occurring in a distributed computing system. Faults may be recorded in a metadata store for verification and analysis by an automated fault management process. Diagnostic procedures may confirm detected faults. The automated fault management process may perform recovery workflows involving operations such as rebooting faulting devices and excommunicating unrecoverable computing nodes from affected clusters.

主权项

1. A distributed database system comprising:
a plurality of computing nodes comprising at least a first subset of the plurality of computing nodes, the first subset configured to perform a distributed computing function, one or more of the plurality of computing nodes configured at least to:
detect a fault involving the first subset of the plurality of computing nodes;perform one or more diagnostic procedures involving at least a component connected to a first computing node of the first subset of the plurality of computing nodes, the one or more diagnostic procedures selected based at least in part on determining that the component is a potential origin of the fault;perform a first one or more operations involving the first computing node, the first one or more operations selected based at least in part on the performing of the one or more diagnostic procedures; andreconfigure the first subset of the plurality of computing nodes to perform the distributed computing function without the first computing node upon determining that performing the first one or more operations has not resolved the fault.