How to failover a node in the event of a double fault
Perform the following operation:
-> AutoSwitchOver = HostFailure | ResourceFailure | Shutdown
If "No" has been set to HaltFlag, a failover is not performed even in the event of a double fault. Setting "Yes" to HaltFlag allows the Shutdown Facility to stop the failed node forcibly (PANIC, power discontinuity, and restart) in the event of a double fault. Then, a failover is performed.
Note
Even though the AutoSwitchOver attribute has been set, a failover is not performed unless HaltFlag has been set in the event of a double fault.
How to failover a userApplication in the event of a node failure, resource failure, and RMS stop
Perform the following operation:
-> AutoSwitchOver = HostFailure | ResourceFailure | Shutdown
Note
In the event of a double fault, a failover is not performed even though this attribute value has been set.
Set the HaltFlag attribute for performing a failover even in the event of a double fault.
When the status of the userApplication to be switched is Fault, it cannot be switched even though AutoSwitchOver has been set.
When performing a failover, clear the Faulted state.
How to start up userApplication automatically when RMS is started
Perform the following operation:
-> AutoStartUp = Yes
If "Yes" has been set to AutoStartUp attribute, the status of a cluster application is automatically transited to Online at RMS startup.
How to switch userApplication to Standby automatically when RMS is started, userApplication is switched, or when clearing a fault state of userApplication
Perform the following operation:
-> StandbyTransitions = Startup | SwitchRequest | ClearFaultRequest
Note
If "Yes" has been set to AutoStartUp attribute, the status of the standby userApplication is transited to Standby when RMS is started regardless of the setting value of StandbyTransitions.
The relationship between AutoStartUp and StandbyTransitions is as follows.
RMS Startup node | AutoStartUp=Yes | AutoStartUp=No | |||
|---|---|---|---|---|---|
StandbyTransitions | StandbyTransitions | ||||
No | StartUP | No | StartUP | ||
Multiple nodes | Operational side uap | Online | Online | Offline | Standby |
Standby side uap | Standby | Standby | Offline | Standby | |
One node only | Standby | Standby | Offline | Standby | |
If the resource which StandbyCapable attribute is set as "yes"(1) does not exist in the userApplication, the userApplication is not in the Standby state regardless of the set value of StandbyTransitions attribute.
How to set scalable cluster applications for preventing timeout of Controller resource during a state transition
When it takes time to start up and stop a cluster application that constitutes a scalable configuration, a timeout error of the Controller resource (resource to indicate the scalability) may occur during a state transition. In this case, the state transition is stopped forcibly.
In this case, the setting of Controller resource needs to be changed according to the startup and stop times for each cluster application that constitutes a scalable configuration.
Calculate the Timeout value of a scalable cluster application, and then change its setting with the following procedure:
Calculating the maximum state transition time for a cluster application
The status of the Controller resource is transited to Online when the statues of userApplications under the Controller resource are all Online. Therefore, calculate the total values of ScriptTimeouts for each resource that configures a cluster application.
For example, if every one of the following resource; Cmdline resource, Fsystem resource, Gds resource, or Gls resource exist under the cluster application, you can calculate as follows. (The timeout value for each resource is a default value.)
Cmdline resource 300 (sec) + Fsystem resource 180 (sec) + Gdsresource 1800 (sec) + Gls resource 60 (sec) = 2340 (sec)
This value is larger than the default value for the scalable cluster application 180 (sec), set the setting value to 2340 (sec).
Information
Default script timeout values for each resource
Cmdline : 300
Fsystem : 180
GDS : 1800
Gls : 60
Considering the number of SysNode
Calculate the considered number of SysNode that configures a cluster application.
The value calculated in Step 1 is the value where the number of SysNode is considerate.
Minus 1 from the number of SysNode and double the value. Then, multiply it by the one calculated in Step 1.
The maximum state transition time of a cluster application between multiple nodes= "1) value" x 2 ( "the number of SysNode" -1 )Example
For example, in the case Online or Offline processing of a userApplication is assumed to be finished just before it times out when the userApplication is with a three-node configuration and the status is Online on Node1, after starting the state transition on the first Node, it takes 4 times (2 x ("the number of SysNode" - 1) for the userApplication to be Online on the final node as follows:
Offline processing on Node1
Online processing on Node2
Offline processing on Node2
Online processing on Node3
Calculating the total values of Step 2 for each cluster application
Setting the calculated value in Step 3 to the timeout value of the Controller resource
See "4.2 Changing timeout period of controller" in "PRIMECLUSTER Reliant Monitor Services (RMS) with Wizard Tools Configuration and Administration Guide" to change the timeout value of the Controller resource.
How to stop a standby operational system preferentially in the event of a heartbeat error
When a heartbeat error is detected, set the survival priority for the node to be stopped forcibly so that it prevents all operational and standby systems from being failed by forcibly stopping both operational and standby systems mutually. Below describes how to stop the operational system preferentially and collect the information for investigation.
Note
The weighting of each node to set in the Shutdown Facility is defined to a node.
If an operational and standby system is switched due to a failover or switchover, it cannot be enabled even though the setting is changed. As before, stop an operational system forcibly after a given time has elapsed in a standby system.
When a cluster is switched, be sure to perform a failback.
If a system panic, CPU load, or I/O load continues, it seems like a heartbeat has an error. In this case, the cluster node with an error is forcibly stopped regardless of the survival priority.
A standby system with a low survival priority waits until an operational system is forcibly stopped completely. During this waiting time, if the heartbeat is recovered, some information for investigating the heartbeat error may not be collected.
This case may occur when the CPU load or I/O load is the high in an operational system.
Below indicates an example when the operational system is node1, and the standby system is node2.
Note
Perform the Steps 1 to 4 in the both operational and standby systems.
Modify the SF configuration (/etc/opt/SMAW/SMAWsf/rcsd.cfg) for the standby system (node2) with the vi editor, and so on to give a higher weight value to the standby system. Change the weight attribute value of node2 from "1" to "2."
node2# vi /etc/opt/SMAW/SMAWsf/rcsd.cfg
[Before edit]
node1,weight=1,admIP=x.x.x.x:agent=SA_xx,timeout=20:agent=SA_yy:timeout=20 node2,weight=1,admIP=x.x.x.x:agent=SA_xx,timeout=20:agent=SA_yy:timeout=20
[After edit]
node1,weight=1,admIP=x.x.x.x:agent=SA_xx,timeout=20:agent=SA_yy:timeout=20 node2,weight=2,admIP=x.x.x.x:agent=SA_xx,timeout=20:agent=SA_yy:timeout=20
Note
Describe the setting of one node with one line in the rcsd.cfg file.
admIP may not be described depending on the version of PRIMECLUSTER.
Restart the SF with the sdtool -r command.
It takes about five seconds to execute the sdtool -r command. After that, the changed SF configuration is reflected to the SF.
node2# sdtool -r
Use the sdtool -C command. to check that the changed SF configuration has been reflected
Check that the weight attribute value of node2 has become "2."
node2# sdtool -C
Cluster Host Type Weight Admin IP Agent List (Agent:timeout)
------------ ----- ------ -------- --------------------------
node1 CORE 1 x.x.x.x SA_xx:20,SA_yy:20
node2 CORE 2 x.x.x.x SA_xx:20,SA_yy:20
Note
"Type" may not be displayed depending on the version of PRIMECLUSTER.
Use the sdtool -s command to check that all the SAs defined to the SF operate properly. Moreover, check that "Test State" and "Init State" have been changed to "TestWorked" and "InitWorked respectively.
node2# sdtool -s
Cluster Host Agent SA State Shut State Test State Init State
------------ ----- -------- ---------- ---------- ----------
node1 SA_xx Idle Unknown TestWorked InitWorked
node1 SA_yy Idle Unknown TestWorked InitWorked
node2 SA_xx Idle Unknown TestWorked InitWorked
node2 SA_yy Idle Unknown TestWorked InitWorkedNote
Perform the following Steps 5 to 8 either in the operational or standby system.
Check the ShutdownPriority attribute value of a cluster application (userApplication) with hvutil -W command.
When the ShutdownPriority attribute value is other than "0," perform Steps 6 to 8.
When it is "0," no more setting is required.
node1# hvutil -W
4Stop PRIMECLUSTER (RMS).
Note
If you stop PRIMECLUSTER (RMS), the operation is also stopped.
node1# hvshut -aChange the ShutdownPriority attribute value of a cluster application (userApplication) to "NONE." First, start the RMS Wizard.
At the Global Cluster Services screen, select userApplication Configuration Wizard.
From the tree on the left of the userApplication Configuration Wizard screen, select the userApplication to be changed, right-click the mouse to display the pop-up menu, and select Edit userApplication or Resource.
Select "ShutdownPriority."
After the attribute is changed, click the Registration button to register it to RMS Configuration.
Click Yes when the message "0817 Do you want to distribute RMS Configuration?" is displayed.
See
For details, see "11.1 Changing the Operation Attributes of a Cluster Application."
Start PRIMECLUSTER (RMS).
node1# hvcm -aNote
When a cluster is switched, be sure to perform a failback.
How to stop the operational node forcibly in the event of a subsystem hang
The following event is called a subsystem hang: the cluster does not detect that the operation is stopped (the operation seems normal from the cluster monitoring) because only some I/Os within the operational node have errors and other I/Os operate normally.
In this case, if the node is switched to a standby node, the operation may be restarted. In the event of a subsystem hang, ping may respond properly and you may be able to log in to a node.
When a subsystem hang is detected, stop the operational node with the following method and switch the operation.
If you can log in to a standby node
Stop the operational node from the standby node with the sdtool command.
# sdtool -k node-name
node-name : CF node name of the operational nodeIf the operational node cannot be forcibly stopped with the sdtool -k command, take the procedures described in "If you cannot log in any node."
If you cannot log in any node
Take any one of the following procedures to stop the node manually and forcibly:
When stopping the physical environment or the control domain forcibly
Refer to the instruction manual of the main unit.
When stopping the guest domain forcibly
Use a command such as the ldm panic-domain command on the control domain.
# ldm panic-domain <guest domain name to be forcibly stopped>
When stopping the Kernel Zone forcibly
Use a command such as the zoneadm halt command on the global zone.
# zoneadm -z <Kernel Zone name to be forcibly stopped> halt
Note
It is possible to determine a subsystem hang from application failures to control a forcible stop mentioned above. In the case, it needs to be determined from multiple clients. That is, even though an error is found from one client, the error may be in the client or on the network. You need to consider such a case when controlling a forcible stop.
How to failover a system when an error occurs the line of LAN
PingHost is necessary information to check the response (ping) with the set host. You should specify two or more nodes that are not configured for the cluster system to prevent adverse effects from hub and router failures.
If PingHost is not set, even if an error occurs in the line of LAN, it does not become a resource error. In this case, the system will not be switched.
Whether PingHost is set or not, if the interface of the takeover IP address is no longer in UP state or a linkdown is detected, the resource becomes abnormal and a switch occurs.
How to set the PreCheckScript to a cluster application
For how to set the PreCheckScript to a cluster application, see "4.1 Setting PreCheckScript in cluster application" in "PRIMECLUSTER Reliant Monitor Services (RMS) with Wizard Tools Configuration and Administration Guide."