Dynamically updating routing information while avoiding deadlocks and preserving packet order after a link error

摘要

A system for allowing dynamic changing of routing information of a network interconnect while avoiding deadlocks and preserving packet ordering. A network resiliency system detects when an error in the network interconnect occurs and dynamically generates new routing information for the routers that factors in the detected error. The network resiliency system then generates new routing information that factors in the failure. The network resiliency system then directs the network interconnect to enter a quiescent state in which no packets are transiting through the network interconnect. After the network interconnect enters the quiescent state, the network resiliency system directs the loading of the new routing information into the routing tables of the network interconnect and then directs the network interconnect to start injecting request packets into the network interconnect.

主权项

1. A method for recovering from a failure in a network interconnect having processors connected via routers, the method comprising:
upon detecting a failure,
requesting a compute routing information component to compute new routing information for the network interconnect;requesting a quiesce component to suppress the injecting of request packets into the network interconnect while allowing response packets to be injected into the network interconnect and in-transit request packets to continue through the network interconnect; andafter the injection of request packets is suppressed,
waiting for the network interconnect to enter a quiescent state in which neither request packets nor response packets are in transit through the network interconnect; andafter the network interconnect enters the quiescent state,
requesting an install routing information component to install the new routing information into the network interconnect; andafter the new routing information is installed into the network interconnect, requesting an unquiesce component to allow the injecting of request packets into the network interconnectwherein each processor is connected to a router through a network interface controller, each processor is connected to a local controller through a network that is out-of-band from the network interconnect, and the suppressing of the injecting of request packets includes the local controller notifying a program executing on the processor to not send requests for injection of request packets into the network interconnect.