PRIMECLUSTER Installation and Administration Guide 4.1 (for Solaris(TM) Operating System)

7.4.1.1 Failure Detection and Cause Identification if a Failure Occurs

If a failure occurs in a resource, you can use the functions of PRIMECLUSTER and the operating system to detect the failure and identify the faulted resource that caused the failure.

The descriptions given in (a) to (k) below are relevant to the "Failure confirmation features list" given below:

Failure detection

Normally, the RMS main window (b) is used to monitor the cluster applications.

• If a failure occurs in a resource or the system

  Failover of the userApplication or node panic will occur.

  In such a case, you can detect the failure by observing the following conditions:

  • A pop-up message screen (a) is displayed.
  • The color of the icons in the RMS main window (b) changes.
  • A message is output to the msg main window (g), Syslog(j), and the console (k).

     

• If a warning-level failure occurs in the system

  If a warning-level failure (for example, insufficient disk space or insufficient swap area) occurs in the system, you can detect the failure by observing the following conditions:

  • The node icon in the CRM main window (d) changes.
  • A message is output to Syslog(j) and the console (k).

     

• If RMS fails to start on all the nodes, the userApplication will not start. You can start the userApplication by executing the "clreply" command.
  • By executing the "clreply" command, you can confirm an operator intervention request to which no response has been entered and start up the userApplication by responding to it. For information on the "clreply" command, see the manual pages.
  • The operator intervention request message will be output to Syslog(j) and the console (k). By responding to the operator intervention request message, you can start the userApplication.

  For further details, see "Operator Intervention Messages."

  

  If there are multiple operator intervention request messages for which no response has yet been entered, you need to respond to each of them.

In addition, you can use the features described in "Failure confirmation features" to detect the failure.

Cause identification

You can also use the function that detected the failure and the features listed in "Failure confirmation features" below to identify the faulted resource that caused the failure.

Failure confirmation features list

Failure confirmation features		Manual reference
(a)	Message screen	"Failed Resource Message"
(b)	RMS main window The RMS tree and the RMS cluster table can be used from this screen.	"RMS Main Window"
(c)	CF main window The CF tree can be used from this screen.	"CF Main Window"
(d)	CRM main window The CRM tree can be used from this screen. This screen is useful in detecting hardware resource faults.	"CRM Main Window"
(e)	"Resource Fault History" screen This screen is useful in detecting hardware resource faults.	"Resource Fault History "
(f)	Current list of resources in which a failure has occurred	"Fault Resource list"
(g)	MSG main window The cluster control messages can be viewed in this screen. To display this screen, select the msg tab in the Cluster Admin screen.	-
(h)	Application log	"Viewing application logs"
(i)	switchlog	"Viewing switchlogs"
(j)	Syslog	-
(k)	Console Messages that are displayed on the console can be checked. Viewing the "console problem" information on the console can help you identify the fault cause.	"Messages"
(l)	Machine management GUI	"Machine Administration Guide."
(m)	MultiPathDisk view	"Multipath Disk Control Load Balance option x.x Guide."
(n)	GDS GUI	"PRIMECLUSTER Global Disk Services Configuration and Administration Guide."