Confined recovery in a distributed computing system

摘要

Executing a confined recovery in a distributed system having a plurality of worker systems including a failed worker system at a current superstep. The confined recovery includes determining states of the partitions of the worker systems during the supersteps preceding the current superstep, and determining a recovery initiation superstep preceding the current superstep in which all messages for recovery initiation superstep are available. The recovery initiation superstep is determined responsive to determining the states of the partitions. Additionally, a recovery set of partitions is determined for which messages in supersteps after the recovery initiation superstep are not available. The worker systems having the partitions in the recovery set are instructed to execute the defined function for the partitions in the recovery set starting at the recovery initiation superstep to recover the lost exchanged messages.

主权项

1. A computer-implemented method for executing a confined recovery in a distributed computing system having a plurality of worker systems, the worker systems executing a computation in a plurality of supersteps, the worker systems having a plurality of partitions executing a defined function during the supersteps that exchange messages with partitions of other worker systems, the method comprising:
identifying a failed worker system from the plurality of worker systems, wherein the failed worker system failed to execute during a current superstep and lost exchanged messages; determining states of the partitions of the worker systems during supersteps preceding the current superstep; determining, responsive to the determination of the states, a recovery initiation superstep preceding the current superstep in which all messages for the recovery initiation superstep are available; determining a recovery set of partitions for which messages in supersteps after the recovery initiation superstep are not available; assigning each partition in the recovery set of partitions to a recovery worker system from a set of recovery worker systems configured to process the partition, wherein the set of recovery worker systems includes a proper subset of the plurality of worker systems; determining messaging worker systems responsible for sending the messages in supersteps after the recovery initiation superstep that are not available, the messaging worker systems being a proper subset of the plurality of worker systems and being different than the recovery worker systems; instructing the recovery worker systems to execute the defined function for the partitions in the recovery set starting at the recovery initiation superstep to recover the lost exchanged messages; and instructing the messaging worker systems to transmit the messages in supersteps after the recovery initiation superstep that are not available to the recovery work systems for use in processing the recovery set of partitions.