A copy of the Resource Database is stored locally on each node in the cluster. When the cluster is up and running, all of the local copies are kept in sync. However, if a node is taken down for maintenance, then its copy of the Resource Database may be out of date by the time it rejoins the cluster. Normally, this is not a problem. When a node joins a running cluster, then its copy of the Resource Database is automatically downloaded from the running cluster. Any stale data that it may have had is thus overwritten.
There is one potential problem. Suppose that the entire cluster is taken down before the node with the stale data had a chance to rejoin the cluster. Then suppose that all the nodes are brought back up again. If the node with the stale data comes up long before any of the other nodes, then its copy of the Resource Database will become the master copy used by all the nodes when they eventually join the cluster.
To avoid this situation, the Resource Database implements a startup synchronization procedure. If the Resource Database is not fully up and running anywhere in the cluster, then starting the Resource Database on a node will cause that node to enter into a synchronization phase. The node will wait up to StartingWaitTime seconds for other nodes to try to bring up their own copies of the Resource Database. During this period, the nodes will negotiate among themselves to see which one has the latest copy of the Resource Database. The synchronization phase ends when either all the nodes have been accounted for or StartingWaitTime seconds have passed. After the synchronization period ends, the latest copy of the Resource Database that was found during the negotiations will be used as the master copy for the entire cluster.
The default value for StartingWaitTime is 60 seconds.
This synchronization method is intended to cover the case where all the nodes in a cluster are down, and then they are all rebooted together. For example, some businesses require high availability during normal business hours, but power their nodes down at night to reduce their electric bill. The nodes are then powered up shortly before the start of the working day. Since the boot time for each node may vary slightly, the synchronization period of up to StartingWaitTime ensures that the latest copy of the Resource Database among all of the booting nodes is used.
Another important scenario in which all the nodes may be booted simultaneously involves the temporary loss and then restoration of power to the lab where the nodes are located.
However, for this scheme to work properly, you must verify that all the nodes in the cluster have boot times that differ by less than StartingWaitTime seconds. Furthermore, you might need to modify the value of StartingWaitTime to a value that is appropriate for your cluster.
Modify the value of StartingWaitTime as follows:
Start up all of the nodes in your cluster simultaneously. It is recommended that you start the nodes from a cold power on. Existing nodes are not required to reboot when a new node is added to the cluster.
After the each node has come up, look in /var/log/messages for message number 2200. This message is output by the Resource Database when it first starts. For example, enter the following command:
# grep 2200 /var/log/messages
Feb 23 19:00:41 fuji2 dcmmond[407]: [ID 888197 daemon.notice] FJSVcluster: INFO: DCM: 2200: Cluster configuration management facility initialization started.Compare the timestamps for the messages on each node and calculate the difference between the fastest and the slowest nodes. This will tell you how long the fastest node has to wait for the slowest node.
Check the current value of StartingWaitTime by executing the clsetparam command on any of the nodes. For example, enter the following command:
# /etc/opt/FJSVcluster/bin/clsetparam -p StartingWaitTime
60The output for our example shows that StartingWaitTime is set to 60 seconds.
If there is a difference in startup times found in Step 2, the StartingWaitTime, or if the two values are relatively close together, then you should increase the StartingWaitTime parameter. You can do this by running the clsetparam command on any one node in the cluster. For example, enter the following command:
# /etc/opt/FJSVcluster/bin/clsetparam -p StartingWaitTime 300This sets the StartingWaitTime to 300 seconds.