PRIMECLUSTER Installation and Administration Guide 4.3

7.4.1 Corrective Action in the event of a resource failure

This section describes the corrective actions to take when a resource failure occurs.

7.4.1.1 Failure Detection and Cause Identification if a Failure Occurs

If a failure occurs in a resource, you can use the functions of PRIMECLUSTER and the operating system to detect the failure and identify the faulted resource that caused the failure.

The descriptions given in (a) to (g) below are relevant to the "Failure confirmation features list" given below:

Failure detection

Normally, the RMS main window (a) is used to monitor the cluster applications.

• If a failure occurs in a resource or the system

  Failover of the userApplication or node panic will occur.

  In such a case, you can detect the failure by observing the following conditions:

  • The color of the icons in the RMS main window (a) changes.

  • A message is output to the msg main window (c), Syslog(f), and the console (g).

    
    

• If a warning-level failure occurs in the system

  If a warning-level failure (for example, insufficient disk space or insufficient swap area) occurs in the system, you can detect the failure by observing the following conditions:

  • A message is output to Syslog(f) and the console (g).

    
    

• If RMS fails to start on all the nodes, the userApplication will not start. You can start the userApplication by executing the "clreply" command.

  • By executing the "clreply" command, you can confirm an operator intervention request to which no response has been entered and start up the userApplication by responding to it. For information on the "clreply" command, see the manual pages.

  • The operator intervention request message will be output to Syslog(f) and the console (g). By responding to the operator intervention request message, you can start the userApplication.

  For further details, see "D.4 Operator Intervention Messages."

  Note

  If there are multiple operator intervention request messages for which no response has yet been entered, you need to respond to each of them.

In addition, you can use the features described in "Failure confirmation features list" to detect the failure.

Cause identification

You can also use the function that detected the failure and the features listed in "Failure confirmation features list" below to identify the faulted resource that caused the failure.

Failure confirmation features list

Failure confirmation features		Manual reference
(a)	RMS main window The RMS tree and the RMS cluster table can be used from this screen.	7.1.3 RMS Main Window
(b)	CF main window The CF tree can be used from this screen.	7.1.1 CF Main Window
(c)	MSG main window The cluster control messages can be viewed in this screen. To display this screen, select the msg tab in the Cluster Admin screen.	-
(d)	Application log	7.3.4.2 Viewing application logs
(e)	switchlog	7.3.4.1 Viewing switchlogs
(f)	Syslog	-
(g)	Console * Messages that are displayed on the console can be checked. Viewing the "console problem" information on the console can help you identify the fault cause.	Appendix D Messages
(h)	GDS GUI	"PRIMECLUSTER Global Disk Services Configuration and Administration Guide."

Note

Console

• The operator intervention request messages (message numbers: 1421, 1423), incurred when RMS is not started on all nodes, are displayed only when yes(1) is set for the AutoStartUp attribute of the userApplication. For information on the userApplication attribute, see"11 Appendix - Attributes" in the "PRIMECLUSTER Reliant Monitor Services (RMS) with Wizard Tools Configuration and Administration Guide."

• The operator intervention request messages (message numbers: 1422, 1423) and the error resource messages incurred after a resource or system error occurs are displayed only when yes(1) is set for the PersistentFault attribute of the userApplication. For information on the userApplication attribute, see "11 Appendix - Attributes" in the "PRIMECLUSTER Reliant Monitor Services (RMS) with Wizard Tools Configuration and Administration Guide."

• The operator intervention request and error resource messages are displayed by using the "clwatchlogd" daemon to monitor switchlog. You need to send the SIGHUP signal to clwatchlogd when you change the value of RELIANT_LOG_PATH that is defined in the "hvenv.local" file. When clwatchlogd receives this signal, clwatchlogd acquires the latest value of RELIANT_LOG_PATH. After you change RELIANT_LOG_PATH, you must start RMS.

Note

When you check the message of a resource failure, a resource with the "MONITORONLY" attribute may be in the fault state even if the cluster application is in the Offline state. Check whether there are any resources in the fault state. Especially, check that Fsystem resources are not in the fault state.

7.4.1.2 Corrective Action for Failed Resources

Take the following steps for failed resources:

1. Correct the faulted resource

   Correct the problem in the failed resource. For details, see "2 Troubleshooting" in "PRIMECLUSTER Reliant Monitor Services (RMS) Reference Guide."

   Note

   If you are using an operation management product other than a PRIMECLUSTER product, you may need to take corrective actions prescribed for that product.

   For details, see the manual provided with each operation management product.

   (Example) Symfoware

2. Recover the cluster application

   At the RMS main window, check the state of the cluster application to which the corrected resource is registered. If the cluster application is in the Faulted state, execute the Fault clear operation.

   For details on the Fault clear operation, see "7.2.2.4 Bringing Faulted Cluster Application to Online State."