Failure and Recovery Mechanisms

Each processor and queue module has attributes that allow the Simulation Kernel to determine if it expects to receive failure and/or recovery interrupts. The attribute failure intrpts indicates whether the module expects to receive failure interrupts and, similarly, the attribute recovery intrpts indicates whether the module expects recovery interrupts.

These attributes may have one of three values: “network wide”, “local only” or “disabled”. If the attribute is set to “network wide”, then the module will receive an interrupt when any node or link fails or recovers, regardless of location in the network. If the attribute is set to “local only”, then the module will receive an interrupt only when its parent node fails or recovers. If the attribute is set to “disabled”, then the module will not receive any failure or recovery interrupts. “Disabled” is the default value for the attributes, because most modules typically do not expect these interrupts.

When a node failure occurs, several related actions take place. The Simulation Kernel suspends the current event and delivers the failure interrupts to the correct modules throughout the model. After all the failure interrupts are delivered, then the Simulation Kernel notifies the node’s transmitter and receiver channels of the failure. All busy transmitter channels abort their current packets in progress and flush their transmit queues. Each busy receiver channel sets the node failure TDA in its packet(s) to the current simulation time, if this is the first time the node failed during each packet’s reception. In the default transceiver pipelines, a packet will not be accepted if the node failed at any time during its transmission or reception. After the interrupts are delivered and the transmitter and receiver channels have completed their actions, the condition attribute value is internally set to “disabled” and the suspended event is reactivated.

After a node recovers from a failure, the condition attribute value is automatically set to “enabled” and then a set of actions occur. The Simulation Kernel suspends the current event and delivers the recovery interrupts to the correct modules. After all the recovery interrupts are delivered, then the Simulation Kernel notifies the node’s receiver channels of the recovery. Each busy receiver channel sets the node recovery TDA in its packet(s) to the current simulation time. After the interrupts have been delivered and the receiver channels have completed their actions, the suspended event is reactivated.

When a link failure occurs, several related actions take place. The Simulation Kernel suspends the current event and delivers the failure interrupts to the correct modules. After all the failure interrupts are delivered, the condition attribute value is automatically set to “disabled” and the suspended event is reactivated. A link failure does not automatically cause packets to be dropped, as is the case of a node failure. However, after a link is disabled, transmitters prevent new packets from being sent on the link.

After a link recovers from a failure, the condition attribute value is automatically set to “enabled”. The Simulation Kernel then suspends the current event and delivers the recovery interrupts to the correct modules. These interrupts are delivered forced, which means that the event that is changing the node’s condition attribute is suspended and each of the recovery interrupts is delivered during the event. After all the recovery interrupts are delivered, the suspended event is reactivated.