Since redundant cluster interconnects are required for reliability, ICF is designed to use all of the available bandwidth. Each connection between two nodes in the cluster is called a route. When ICF has a message to send to another node, it chooses a route so that the message traffic is balanced between all the available routes. In the four-node cluster shown in Figure 3.2 Typical four-node cluster, each node has two routes to each of the other nodes.
When everything is functioning properly in the cluster, all of the routes are in the UP state (available for use). Every UP route participates in carrying message traffic if it functions at the same speed (mixed speed cluster interconnects are discussed below.) In addition, every route is always used for heartbeat requests. (A heartbeat is a message indicating that a node is functional.) If a certain number of heartbeat requests fail on a route, then that route is marked as DOWN. A DOWN route is never used for message traffic. However, heartbeat requests are still attempted on the DOWN routes, and if one succeeds, the route is returned to the UP state. This last behavior is sometimes called self-healing routes. The use of all the UP routes for message traffic is called port aggregation or trunking.
Failures of heartbeat requests can come from several causes. When a network component fails, that route is not usable and the behavior is the same as described in the previous paragraph. When a node is DOWN to the only remaining route to another node, PRIMECLUSTER does not mark this route as DOWN, rather it marks the node as LEFTCLUSTER.
LEFTCLUSTER is a node state that indicates that the node cannot communicate with other nodes in the cluster. That is, the node has left the cluster. Once a node is marked as LEFTCLUSTER or DOWN, no more heartbeat attempts are made on any of the routes to the node until the node rejoins the cluster.
Note
The last route to a node is never marked DOWN, even if the node is in the LEFTCLUSTER or DOWN state.
Sometimes, events happen on a node that prevents the node from responding to heartbeat requests for some reason other than a failure. For example, a Fibre Channel controller can prevent other network device drivers from executing while attempting a link recovery. This can result in a false detection of node failure. To allow for some flexibility in responding to this condition, the number of missed heartbeat requests on the final route to a node before declaring the node as LEFTCLUSTER is tunable.
You can only adjust the number of failed-heartbeat requests that occur before declaring the node LEFTCLUSTER. The other parameters for ICF are not tunable. It is important that the route-detection algorithm marks routes as down as soon as possible, so that messages are switched to alternate routes without a noticeable delay. If a route is momentarily not available, for whatever reason, the self-healing mechanism will quickly reactivate the route after it becomes functional. "PRIMECLUSTER Cluster Foundation (CF) Configuration and Administration Guide" has details on tuning the ICF parameters.
PRIMECLUSTER also supports the use of non-symmetric cluster interconnects, for example, one cluster interconnect could use gigabit Ethernet and a second fast-Ethernet. When making decisions about routing, the cluster interconnect speed is also considered. In the previous example, the gigabit Ethernet would always be used in preference to the fast-Ethernet whenever it is available. If there are multiple cluster interconnects at the same speed, the port aggregation is done across all the devices with the same speed. Heartbeat requests are always sent on every cluster interconnect, independent of the cluster interconnect speed.
