5.2.4 Fatal error messages

Advertisement server: Data received will be discarded due to receive error on socket. errno = errno

Content:

The shutdown daemon failed to send data by the network communication. When this message is output, an unintended node may be forcibly stopped when the nodes are forcibly stopped.

Corrective action:

Record this message and collect information for an investigation. Then, contact field engineers. For details on how to collect information, see "Troubleshooting" in "PRIMECLUSTER Installation and Administration Guide."

Agent Shutdown Agent uninitialization for host nodename failed

Content:

The shutdown agent is not uninstalled properly.

Corrective action:

Check logs of the shutdown agent and then contact field engineers.

cannot determine the port on which the advertisement server should be started

Content:

A network communication port number for the shutdown daemon is not obtained successfully.

Corrective action:

Execute the following command.

# /opt/SMAW/bin/sdtool -s

When "The RCSD is not running" is displayed, execute the following command to start the shutdown facility.

# /opt/SMAW/bin/sdtool -b

Execute the following command and check the result.

# /opt/SMAW/bin/sdtool -s

The initial status is InitWorked and the test status is TestWorked
No corrective action is required.
The initial status is InitFailed or the test status is TestFailed
Check that following line is configured for the /etc/inet/services file on Solaris, and for /etc/services file on Linux.
```
sfadv           2316/udp                        # SMAWsf package
```
If the line is not configured, add this line to the above file. Take the following actions after correcting the file.
- Solaris
  Execute the following commands to restart the shutdown facility.
```
# /opt/SMAW/bin/sdtool -e
```
```
# /opt/SMAW/bin/sdtool -b
```
- Linux
  Restart the cluster node after correcting the file.
Other than above
Wait for a while and then execute the above command again. Check the result.

If the process does not work as a corrective action, record this message and collect information for an investigation. Then, contact field engineers. For details on how to collect information, see "Troubleshooting" in "PRIMECLUSTER Installation and Administration Guide."

Could not correctly read the rcsd.cfg file.

Content:

Either rcsd.cfg file does not exist or the syntax in rcsd.log is incorrect.

Corrective action:

Create rcsd.cfg file or correct the syntax error.

/etc/sysconfig/libvirt-guests is not configured on Hypervisor of host nodename. rcsd died abnormally.

Content:

The /etc/sysconfig/libvirt-guests on the hypervisor on the node <nodename> is not configured. The shutdown daemon (rcsd) is ended abnormally.

Corrective action:

Take the corrective action depending on the following events.

Event 1

This message is output due to the shutdown or restart of the hypervisor, and the node was not forcibly stopped successfully.

Check if the forcibly stopped guest OS is in LEFTCLUSTER state. Do not recover LEFTCLUSTER state at this time.
Wait until the activated guest OS becomes LEFTCLUSTER state.
Check that all the nodes become LEFTCLUSTER state and then check if the /etc/sysconfig/libvirt-guests is properly configured on all the hypervisors. See "PRIMECLUSTER Installation and Administration Guide" for configuration of the /etc/sysconfig/libvirt-guests.
Stop all the guest OSes on which cluster applications will not be Online state.
Check that all the guest OSes are stopped and then execute the following command on the guest OSes on which the cluster applications will be Online state. This process will recover LEFTCLUSTER state.
```
# cftool -k
```
To start the shutdown facility, execute the following command on the guest OSes on which the cluster applications will be Online state.
```
# /opt/SMAW/bin/sdtool -b
```

Event 2

Other than the above event 1

Check if the /etc/sysconfig/libvirt-guests on the hypervisor on the node <nodename> is properly configured. See "PRIMECLUSTER Installation and Administration Guide" for configuration of the /etc/sysconfig/libvirt-guests.
Execute the following command to start the shutdown facility.
```
# /opt/SMAW/bin/sdtool -b
```
If the process does not work as a corrective action, record this message and collect information for an investigation. Then, contact field engineers. For details on how to collect information, see "Troubleshooting" in "PRIMECLUSTER Installation and Administration Guide."

Failed to create a signal handler for SIGCHLD
Failed to create a signal handler for SIGUSR1

Content:

An internal error occurred in the program.

Corrective action:

Failed to unlink/create/open CLI Pipe

Content:

An internal error occurred in the program.

Corrective action:

Failed to open CFSF device, reason (value)string

Content:

The CFSF device cannot be opened.

Corrective action:

Fail to post LEFTCLUSTER event:string

Content:

When rci detected a node error, it failed in transmission of LEFTCLUSTER event.

Corrective action:

FATAL: rcsd died too frequently.It will not be started by rcsd_monitor.

Content:

The shutdown daemon (rcsd) cannot be restarted after it abnormally ended.

Corrective action:

Check if error messages of the shutdown facility have been output before this message is output. If the error messages are output, follow them for the corrective action.

When no error messages of the shutdown facility are output, record this message and collect information for an investigation. Then, contact field engineers. For details on how to collect information, see "Troubleshooting" in "PRIMECLUSTER Installation and Administration Guide."

fopen of /etc/opt/SMAW/SMAWsf/rcsd.cfg failed, errno errno

Content:

The shutdown facility cannot be started because it has not been configured.

Corrective action:

Check again the configuration of the shutdown facility and then start the shutdown daemon again.

This action is not required for the xen kernel environment on Red Hat Enterprise Linux 5 (for Intel64).

Forced to re open rcsd net pipe due to an invalid pipe name

Content:

The network communication pipe of the shutdown daemon had an error. It has been recreated.

Corrective action:

The shutdown facility recreates the network communication pipe that had an error. No corrective action is required because this error does not affect other processes that are operated by the system.

Forced to re-open rcsd net pipe due to a missing pipe name

Content:

The network communication pipe of the shutdown daemon had an error. It has been recreated.

Corrective action:

The shutdown facility recreates the network communication pipe that had an error. No corrective action is required because this error does not affect other processes that are operated by the system.

Forced to re-open rcsd net pipe due to failed stat pipe name errno: errno

Content:

The network communication pipe of the shutdown daemon had an error. It has been recreated.

Corrective action:

The shutdown facility recreates the network communication pipe that had an error. No corrective action is required because this error does not affect other processes that are operated by the system.

function of file failed, errno errno

Content:

An internal error occurred in the program.

Corrective action:

Check if related messages are additionally output.
If additional messages are output, follow them for the corrective action.

If no additional messages were output, record this message and collect information for an investigation. Then, contact field engineers. For details on how to collect information, see "Troubleshooting" in "PRIMECLUSTER Installation and Administration Guide."

h_cfsf_get_leftcluster() failed. reason: (value)string

Content:

Failed to call cfsf_get_leftcluster.

Corrective action:

HostList empty

Content:

The rcsd.cfg is not configured properly or the rcsd.cfg is not read successfully.

Corrective action:

Execute the following command.

# /opt/SMAW/bin/sdtool -s

When "The RCSD is not running" is displayed, execute the following command to start the shutdown facility.

# /opt/SMAW/bin/sdtool -b

Execute the following command and check the result.

# /opt/SMAW/bin/sdtool -s

The initial status is InitWorked and the test status is TestWorked
No action is required.
The initial status is InitFailed or the test status is TestFailed
Check the configuration of the rcsd.cfg and correct the configuration if it has errors. Then, execute the following commands to restart the shutdown facility.
```
# /opt/SMAW/bin/sdtool -e
```
```
# /opt/SMAW/bin/sdtool -b
```
Other than the above status
Wait for a while and then execute the above commands again. Check the result.

Host <nodename > ICF communication failure detected

Content:

The rcsd was notified that the node <nodename > has lost its heartbeat.

Host nodename MA_exec: string failed, errno errno

Content:

An error has occurred while the MA executing thread for the node <nodename> was executed.

Corrective action:

Malloc failed during function

Content:

Not enough memory.

Corrective action:

Increase virtual memory size (ulimit -v) or increase system memory. If the problem still remains, contact field engineers.

Node id number ICF communication failure detected

Content:

CF layer has detected the lost heartbeat.

Corrective action:

The rcsd will take the action.

rcsd died abnormally. Restart it.

Content:

After an abnormal termination, the shutdown daemon (rcsd) has restarted.

Corrective action:

As the shutdown daemon automatically recovers by itself, no action is required.

SA SA_blade to test host nodename failed
SA SA_ipmi to test host nodename failed
SA SA_lkcd to test host nodename failed

Content:

The node on which the message is displayed failed to connect BMC (iRMC) of the node nodename or the blade server.

Corrective action:

Check the following.

Check if the system or the network has high load.
If this message is no longer displayed after 10 minutes, the error may have been solved. Execute the following command to check if the shutdown facility works properly.
```
# /opt/SMAW/bin/sdtool -s
```

<SA_ipmi is output>

Check if the following are valid: the IP address of BMC that is specified in the /etc/opt/SMAW/SMAWsf/SA_ipmi.cfg, and the user name and password to log in BMC.
See "PRIMECLUSTER Installation and Administration Guide" for 'timeout' configuration to specify the /etc/opt/SMAW/SMAWsf/ SA_ipmi.cfg.
Check if BMC is configured properly.
For how to configure BMC and how to check the configuration, see "User's Guide" of each model or "ServerView User's Guide."
Check if BMC is turned on.
Check if the normal lamp of the port which is connected by HUBs and LAN cables is on.
Check if LAN cables are properly connected to the BMC connector or the HUB connector.
Check if the IP address of BMC belongs to the same segment as the cluster node.
Check if BMC has not restarted or updated the firmware.
Make sure that the LAN access privilege of the user for logging in to BMC specified in SA_ipmi.cfg is set as Administrator.

<SA_blade is output>

Check if the IP address of the management blade and the slot number of the server blade, which are specified in the /etc/opt/SMAW/SMAWsf/SA_blade.cfg, are valid.
See "PRIMECLUSTER Installation and Administration Guide" for 'timeout' configuration to specify the /etc/opt/SMAW/SMAWsf/ SA_ipmi.cfg.
Check if the blade server is configured properly.
For the special notes to configure the shutdown facility, see "PRIMECLUSTER Installation and Administration Guide."
For how to configure the blade server and how to check the configuration, see "ServerView User's Guide" and each hardware guide provided with a machine.
Check if the normal lamp of the port which is connected by HUBs and LAN cables is on.
Check if LAN cables are properly connected to the management blade connector or the HUB connector.
Check if the management blade has not restarted or updated the firmware, or check if the management blade in the master mode has not been switched.

<SA_ lkcd is output>

Check if the timeout value of SA_lkcd that is specified in the /etc/opt/SMAW/SMAWsf/rcsd.cfg is correct.
Check if the /etc/opt/SMAW/SMAWsf/SA_lkcd.tout is also specified properly. To specify these files, see "PRIMECLUSTER Installation and Administration Guide." Correct the errors if needed. Check and correct the errors on all the nodes.
Check if the following command is executed when the shutdown agent is configured.
```
# /etc/opt/FJSVcllkcd/bin/panicinfo_setup
```
To use the Diskdump shutdown agent or the Kdump shutdown agent, execute the above command after the IMPI shutdown agent or the Blade shutdown agent is configured. When the configuration of the IMPI shutdown agent or the Blade shutdown agent is changed, execute the following command again.
When the Diskdump shutdown agent is used, check if Diskdump is configured properly. When the Kdump shutdown agent is used, check if Kdump is configured properly. For how to configure Diskdump or kdump and how to check the configuration, see "PRIMECLUSTER Installation and Administration Guide." Correct the errors if needed. Check and correct the errors on all the nodes.
When the IPMI shutdown agent is used, check the above <SA_ipmi is output>. When the Blade shutdown agent is used, check the above <SA_blade is output>.

If the error cause was any one of the above check items, restart the SNMP asynchronous monitoring function and the shutdown facility by executing the following commands on the node which outputted the above message.

# /opt/SMAW/bin/sdtool -e

# /opt/SMAW/bin/sdtool -b

When the connection fails again even after the above action is taken, a network failure or a failure of either BMC (iRMC), blade server, or HUB may be the cause of the error. In this case, contact field engineers.

SA SA_icmp to test host nodename failed

Content:

The node on which the message is displayed failed to check the connection to the node nodename

Corrective action:

Check the following:

If events such as panic or hang-up has not occurred on the host OS.
If guest OSes, the host OS, or the network do not have high load.
If this message is no longer displayed after 10 minutes, the error may have been solved. Execute the following command to check if the shutdown facility works properly.
```
# /opt/SMAW/bin/sdtool -s
```
If the node nodename is not stopped.
If the IP address or the network interface that is specified for /etc/opt/SMAW/SMAWsf/SA_icmp.cfg is valid.
See "PRIMECLUSTER Installation and Administration Guide" to specify the /etc/opt/SMAW/SMAWsf/SA_icmp.cfg.
If the virtual IP addresses or the network interface that are allocated for guest OSes are valid.
If the normal lamp of the port which is connected by HUBs and LAN cables is on.
If the cables are properly connected.
If the error cause was any one of the above check items, restart the SNMP asynchronous monitoring function and the shutdown facility by executing the following commands on the node which outputted the above message.
```
# /opt/SMAW/bin/sdtool -e
```
```
# /opt/SMAW/bin/sdtool -b
```

When the connection fails again even after the above action is taken, a network failure or a failure of HUB may be the cause of the error. In this case, contact field engineers.

SA SA_ilomp.so to test host nodename failed
SA SA_ilomr.so to test host nodename failed
SA SA_rccu.so to test host nodename failed
SA SA_xscfp.so to test host nodename failed
SA SA_xscfr.so to test host nodename failed

Content:

The node on which the message is displayed failed to check the connection to XSCF, RCCU, or ILOM on the node nodename.

Corrective action:

Check the following.

If the system or the network has high load.
- Solaris PRIMECLUSTER 4.2A00 or earlier
  Execute the following commands to restart both the asynchronous monitoring function for consoles and the shutdown facility.
```
# /opt/SMAW/bin/sdtool -e
# /etc/opt/FJSVcluster/bin/clrccumonctl stop
# /etc/opt/FJSVcluster/bin/clrccumonctl start
# /opt/SMAW/bin/sdtool -b
```
- Solaris PRIMECLUSTER 4.3A10 or later
- If this message is no longer displayed after 10 minutes, the error may have been solved. Execute the following command to check if the shutdown facility works properly.
```
# /opt/SMAW/bin/sdtool -s
```

If the console information of RCCU such as the IP address or the node name is correct.
Use clrccusetup(1M) to check the configured console information. If there is any error in the information, use clrccusetup(1M) to register the console information again.
If RCCU is configured properly.
For how to configure RCCU and how to check the configuration, see an operation manual provided with RCCU.
If RCCU is turned on.
If the normal lamp of the port which is connected by HUBs and LAN cables is on.
If LAN cables are properly connected to the RCCU connector or the HUB connector.
If the IP address of RCCU belongs to the same segment as the administrative LAN.

If the console information of XSCF such as the IP address or the connection protocols (telnet and SSH) is correct.
Use clrccusetup(1M) to check the configured console information. If there is any error in the information, use clrccusetup(1M) to register the console information again.
If XSCF is configured properly.
For the special notes to configure the shutdown facility, see "PRIMECLUSTER Installation and Administration Guide."
For how to configure XSCF and how to check the configuration, see "XSCF (eXtended System Control Facility) User's Guide."
When SSH is used to connect to XSCF
- Check if the login user account for the shutdown facility is used to connect the cluster node with XSCF by SSH, and the user authentication (such as creating RSA key) has been completed to connect by SSH for the first time.
- Check if a password authentication is used as a user authentication to connect the cluster node with XSCF by SSH.
  If an auto password authentication such as a public key authentication is used for XSCF, disable it. For how to configure XSCF or how to check the configuration, see "XSCF (eXtended System Control Facility) User's Guide."
If the normal lamp of the port which is connected by HUBs and LAN cables is on.
If LAN cables are properly connected to the XSCF-LAN port connector of XSCF or the HUB connector.
If the XSCF shell port, which belongs to the telnet port, is not connected from out of a cluster.
Connect to the XSCF shell port through a serial port (tty-a) to check the connection status. For how to connect and how to check the connection, see "XSCF (eXtended System Control Facility) User's Guide."
If the IP address of XSCF belongs to the same segment as the administrative LAN or the asynchronous monitoring sub-LAN.
If the firmware in XSCF has not been restarted or updated, or if an event such as XSCF failover has not occurred.

If the console information of ILOM such as the IP address or the node name is correct.
Use clrccusetup(1M) to check the configured console information. If there is any error in the information, use clrccusetup(1M) to register the console information again.
If ILOM is configured properly.
For the special notes to configure the shutdown facility, see "PRIMECLUSTER Installation and Administration Guide." See below manuals for how to configure ILOM and how to check the configuration.
- ILOM 2.x
  "Integrated Lights Out Manager User's Guide"
- ILOM 3.0
  "Integrated Lights Out Manager (ILOM) 3.0 Concepts Guide"
  "Integrated Lights Out Manager (ILOM) 3.0 Web Interface Procedures Guide"
  "Integrated Lights Out Manager (ILOM) 3.0 CLI Procedures Guide"
  "Integrated Lights Out Manager (ILOM) 3.0 Getting Started Guide"
Check if the login user account for the shutdown facility is used to connect the cluster node with ILOM by SSH, and the user authentication (such as creating RSA key) has been completed to connect by SSH for the first time.
For ILOM 3.0, check if a password authentication is used as a user authentication to connect the cluster node with ILOM by SSH.
If an auto password authentication such as a public key authentication is used for ILOM, disable it. For how to configure ILOM or how to check the configuration, see the above manuals for ILOM 3.0.
If the normal lamp of the port which is connected by HUBs and LAN cables is on.
If LAN cables are properly connected to network control (NET MGT) port connector of ILOM or the HUB connector.
If ILOM has not restarted or updated the firmware.

# /opt/SMAW/bin/sdtool -e

# /etc/opt/FJSVcluster/bin/clrccumonctl stop

# /etc/opt/FJSVcluster/bin/clrccumonctl start

# /opt/SMAW/bin/sdtool -b

When the connection fails again even after the above action is taken, a network failure or a failure of either RCCU, XSCF, ILOM, or HUB may be the cause of the error. In this case, contact field engineers.

SA SA_kzchkhost to test host nodename failed

Content:

The node on which the message is displayed failed to check the connection to ILOM or XSCF on the node nodename.

Corrective action:

Check the following points.

If the system or the network has high load.
If this message is no longer displayed after 10 minutes, the error may have been solved. Execute the following command to check if the shutdown facility works properly.
```
# /opt/SMAW/bin/sdtool -s
```
If the normal lamp of the port which is connected by HUBs and LAN cables is on.

If the global zone host information such as the IP address or node name of ILOM is correct.
Use the SF Wizard to check the set global zone host information.
If there is any error in the global zone host information, use the SF Wizard to register the information again.
If ILOM is configured properly.
For the special notes to configure the shutdown facility, see "PRIMECLUSTER Installation and Administration Guide."
See below manuals for how to configure ILOM and how to check the configuration:
"Integrated Lights Out Manager (ILOM) 3.0 Concepts Guide"
"Integrated Lights Out Manager (ILOM) 3.0 Web Interface Procedures Guide"
"Integrated Lights Out Manager (ILOM) 3.0 CLI Procedures Guide"
"Integrated Lights Out Manager (ILOM) 3.0 Getting Started Guide"
Check if the login user account for the shutdown facility is used to connect Kernel Zone with ILOM by SSH, and the user authentication (such as creating RSA key) has been completed to connect by SSH for the first time.
Check if a password authentication is used as a user authentication to connect Kernel Zone with ILOM by SSH.
If an auto password authentication such as a public key authentication is used for ILOM, disable it. For how to configure ILOM or how to check the configuration, see the above manuals for ILOM 3.0.
If LAN cables are properly connected to network control (NET MGT) port connector of ILOM or the HUB connector.
If ILOM has not restarted or updated the firmware.

If the configuration information of the logical domains has been saved.
If the states of the logical domains can be checked in XSCF.
When either of the above is found to be the cause of the error, use the ldm add-spconfig command to save the confirmation information of the logical domains.
If the system or the network has high load.
If the global zone host information such as the IP address, node name, or the connection protocols (telnet and SSH) in XSCF is correct.
Use the SF Wizard to check the set global zone host information.
If there is any error in the global zone host information, use the SF Wizard to register the information again.
If XSCF is configured properly.
For the special notes to configure the shutdown facility, see "PRIMECLUSTER Installation and Administration Guide."
For how to configure XSCF and how to check the configuration, see "Fujitsu SPARC M12 and Fujitsu M10/SPARC M10 System Operation and Administration Guide."
When SSH is used to connect to XSCF.
- Check if the login user account for the shutdown facility is used to connect Kernel Zone with XSCF by SSH, and the user authentication (such as creating RSA key) has been completed to connect by SSH for the first time.
- Check if a password authentication is used as a user authentication to connect Kernel Zone with XSCF by SSH.
  If an auto password authentication such as a public key authentication is used for XSCF, disable it.
  For how to configure XSCF and how to check the configuration, see "Fujitsu SPARC M12 and Fujitsu M10/SPARC M10 System Operation and Administration Guide."
If LAN cables are properly connected to the XSCF-LAN port connector of XSCF or the HUB connector.
If the XSCF shell port is not connected from out of a cluster.
Connect to the XSCF shell port through a serial port to check the connection status. For how to connect and how to check the connection, see "Fujitsu SPARC M12 and Fujitsu M10/SPARC M10 System Operation and Administration Guide."
If the IP address of XSCF belongs to the same segment as the administrative LAN or the asynchronous monitoring sub-LAN.
If the firmware in XSCF has not been restarted or updated, or if an event such as XSCF failover has not occurred.

If the error cause was any one of the above check items, restart the shutdown facility by executing the following commands on the node which outputted the above message.

# /opt/SMAW/bin/sdtool -e

# /opt/SMAW/bin/sdtool -b

When the connection fails again even after the above action is taken, a network fault or a hardware failure of ILOM, XSCF, or HUB may be the error cause. In this case, contact field engineers.

If the problem is not resolved by the above action, record this message, collect information for an investigation, and then contact field engineers.

SA SA_kzonep to test host nodename failed
SA SA_kzoner to test host nodename failed

Content:

The node on which the message is displayed failed to check the connection to Kernel Zone on the node nodename.

Corrective action:

Check the following points:

If the system or the network has high load.
If this message is no longer displayed after 10 minutes, the error may have been solved. Execute the following command to check if the shutdown facility works properly.
```
# /opt/SMAW/bin/sdtool -s
```
If the normal lamp of the port which is connected by HUBs and LAN cables is on.
If LAN cables are properly connected to the administrative LAN port connector or the HUB connector.
If the Kernel Zone information such as the zone name or the global zone host name is correct.
Use the SF Wizard to check the set Kernel Zone information.
If there is any error in the Kernel Zone information, use the SF Wizard to register the information again.
Check if the login user account for the shutdown facility is used to connect Kernel Zone with the global zone host by SSH, and the user authentication (such as creating RSA key) has been completed to connect by SSH for the first time.
If XSCF is configured properly.
For the special notes to configure the shutdown facility, see "PRIMECLUSTER Installation and Administration Guide."

If the error cause was any one of the above check items, restart the shutdown facility by executing the following commands on the node which outputted the above message.

# /opt/SMAW/bin/sdtool -e

# /opt/SMAW/bin/sdtool -b

When the connection fails again even after the above action is taken, a network fault or a hardware failure of HUB may be the error cause. In this case, contact field engineers.

If the problem is not resolved by the above action, record this message, collect information for an investigation, and then contact field engineers.

SA SA_libvirtgp to test host nodename failed
SA SA_libvirtgr to test host nodename failed

Content:

The node on which the message is displayed failed to check the connection to the hypervisor on the node nodename.

Corrective action:

Check the following.

If the guest OSes, hypervisor, or the network has high load.
If this message is no longer displayed after 10 minutes, the error may have been solved. Execute the following command to check if the shutdown facility works properly.
```
# /opt/SMAW/bin/sdtool -s
```
If the following configurations are correct.
- /etc/opt/SMAW/SMAWsf/SA_libvirtgp.cfg
- /etc/opt/SMAW/SMAWsf/SA_libvirtgr.cfg
Specify a user password for the shutdown facility that is encrypted by the /opt/SMAW/bin/sfcipher command.
See "PRIMECLUSTER Installation and Administration Guide" to specify these files.
If a passphrase is not configured to connect to the hypervisor.
If the hypervisor or the guest OSes are configured properly.
If the virtual IP addresses that are allocated for guest OSes are valid.
See "PRIMECLUSTER Installation and Administration Guide" for configuration of the hypervisor or the guest OSes.
If the sudo command is configured for a login user account of the shutdown facility.
See "PRIMECLUSTER Installation and Administration Guide" for configuration of the sudo command.
Check if the login user account for the shutdown facility is used to connect the guest OS (node) with hyper visor by SSH, and the user authentication (such as creating RSA key) has been completed to connect by SSH for the first time.
Check If a password authentication is used as a user authentication to connect the guest OS (node) with hyper visor by SSH.
If an auto password authentication such as a public key authentication is used for hyper visor, disable it. If the normal lamp of the port which is connected by HUBs and LAN cables is on.
If the cables are connected properly.
If events such as panic or hang-up has not occurred on the hyper visor.

# /opt/SMAW/bin/sdtool -e

# /opt/SMAW/bin/sdtool -b

When the connection fails again even after the above action is taken, hardware failure such as a network failure or a failure of HUB may be the cause of the error. In this case, contact field engineers.

SA SA_mmbp.so to test host nodename failed
SA SA_mmbr.so to test host nodename failed

Content:

The node on which the message is displayed failed to check the connection to MMB on the node nodename.

Corrective action:

When this message is output only right after the OS is started and the below 3083 message is output 10 minutes after the message has been output, no corrective action is required. The message is displayed only because the snmptrapd daemon is under activation.

3083: Monitoring another node has been started.

When the actions above do not work, check the following settings.

If the IP address of MMB is properly specified. Check also if the IP address of the administrative LAN on the local node, which is configured for the shutdown daemon of the MMB shutdown facility, is properly specified.
If the user name and the password to control MMB by RMCP, which are registered for the MMB information of the MMB shutdown facility, are valid.
If the user's [Privilege] to control MMB by RMCP, which is configured for the MMB shutdown facility, is "Admin".
If the user's [Status] to control MMB by RMCP, which is configured for the MMB shutdown facility, is "Enabled ".

For how to check the configuration to control MMB by RMCP, which is configured for the MMB shutdown facility, see the manual provided with the machine.

If the IP address of MMB is changed, or the MMB information of the MMB shutdown facility should be changed because it has an error, see the description of changing the administration configuration in "PRIMECLUSTER Installation and Administration Guide" for the corrective action. If the IP address of MMB is changed, the procedure to change the IP address of MMB, which is described in the manual, is not required.

If other configurations have errors, correct them and then execute the following commands on all the nodes to restart the shutdown facility.

# /opt/SMAW/bin/sdtool -e

# /opt/SMAW/bin/sdtool -b

If the configuration does not have errors, check the following:

If the snmptrapd daemon is started (check the process of snmptrapd by the ps [1] command or other commands).
If the normal lamp of the port which is connected by HUBs and LAN cables is on.
If LAN cables are properly connected to the MMB port connector of XSCF or the HUB connector.
If MMB has not restarted or updated the firmware, or if an event such as a switch of Active MMB has not occurred.
If MMB does not have failures.
If the nodes and the network do not have load.

If either of the above is found to be the cause of the error, the MMB asynchronous monitoring will be automatically recovered after the corrective action is taken. Maximum 10 minutes are required for this automatic recovery.

SA SA_pprcip.so to test host nodename failed
SA SA_pprcir.so to test host nodename failed

Content:

The node on which the message is displayed failed to check the connection to RCI on the node nodename.

Corrective action:

Check the following:

If the system does not have load.
- Solaris PRIMECLUSTER 4.2A00 or earlier
  Execute the following commands to restart the RCI asynchronous monitoring function and the shutdown facility.
```
# /opt/SMAW/bin/sdtool -e
# /etc/opt/FJSVcluster/bin/clrcimonctl stop
# /etc/opt/FJSVcluster/bin/clrcimonctl start
# /opt/SMAW/bin/sdtool -b
```
- Solaris PRIMECLUSTER 4.3A10 or later
- If this message is no longer displayed after 10 minutes, the error may have been solved. Execute the following command to check if the shutdown facility works properly.
```
# /opt/SMAW/bin/sdtool -s
```
If RCI is properly configured. The RCI address does not have following errors.
- If the RCI address is configured.
- If the RCI address is not duplicated.
- If the RCI address of another node is changed while the RCI asynchronous monitoring function is active.
If the cable is connected properly.
The monitoring time out time through SCF/RCI is properly configured in the /etc/system file. For how to configure the monitoring timeout time, see " PRIMECLUSTER Installation and Administration Guide"
If the RCI device has not restarted or upgraded the firmware.

# /opt/SMAW/bin/sdtool -e

# /etc/opt/FJSVcluster/bin/clrcimonctl stop

# /etc/opt/FJSVcluster/bin/clrcimonctl start

# /opt/SMAW/bin/sdtool -b

When the connection fails again even after the above action is taken, hardware failure such as a failure of RCI cables or a failure of the System Control Facility (SCF) may be the cause of the error. In this case, record this message and collect both the SCF dump and information for investigation before contacting our field engineers. For how to collect the SCF dump and information for investigation, see "PRIMECLUSTER Installation and Administration Guide"

SA SA_rpdu to test host nodename failed

Content:

Connection from the node where the error message is output to the Remote Power Distribution Unit connected to the unit of the node nodename failed.

Corrective action:

Check the following points:

Corrective action 1

Check if any error occurred in the Remote Power Distribution Unit.
See the manual of the Remote Power Distribution Unit to check if any error occurred in the Remote Power Distribution Unit. Follow the manual and take the corrective action.

Corrective action 2

The system or the network is not under load.
If this message is no longer displayed 10 minutes after the first message is displayed, the connection may be recovered.
Execute the following command to check that the Shutdown Facility is working correctly.
```
# /opt/SMAW/bin/sdtool -s
```

Check if the following items of the Remote Power Distribution Unit which is specified in the RPDU Shutdown Agent are set correctly: IP address, outlet number, user name, and password.
Check the setting of the RPDU Shutdown Agent.
For details, see "PRIMECLUSTER Installation and Administration Guide."

The outlet connected to the node is ON.
Execute the following command to check the status of the Remote Power Distribution Unit.
```
# /opt/SMAW/SMAWsf/bin/sfrpdupoweron -l
```
If the displayed status of the outlet is OFF, execute the following command to switch the outlet to ON.
```
# /opt/SMAW/SMAWsf/bin/sfrpdupoweron -p CF nodename
```

The normal lamp of the port connected to HUB and LAN cable is ON.
The LAN cable is correctly connected to the HUB connector or the LAN port connector of the Remote Power Distribution Unit.
The firmware is not restarted nor updated in the Remote Power Distribution Unit.
The user who accesses the Remote Power Distribution Unit has the required authority.
The required authority may not be given to the user specified for the setting of the RPDU Shutdown Agent.
For details, see "PRIMECLUSTER Installation and Administration Guide."

If any of the status above is the cause of the connection failure, take the corrective action first.
After that, execute the following commands on the node where the message is output, and then restart the Shutdown Facility.

# /opt/SMAW/bin/sdtool -e

# /opt/SMAW/bin/sdtool -b

If the above corrective actions cannot solve this error, record this message and collect information for an investigation. Then, contact field engineers.

For details on how to collect information, see "Troubleshooting" in "PRIMECLUSTER Installation and Administration Guide."

SA SA_sunF to test host nodename failed

Content:

The node on which the message is displayed failed to check the connection to ALOM on the node nodename.

Corrective action:

Check the following:

If the system does not have load.
If this message is no longer displayed after 10 minutes, the error may have been solved. Execute the following command to check if the shutdown facility works properly.
```
# /opt/SMAW/bin/sdtool -s
```
If the IP address of ALOM, the user name and the password to log in ALOM, which are specified in the /etc/opt/SMAW/SMAWsf/SA_sunF.cfg are correct.
For how to specify the /etc/opt/SMAW/SMAWsf/SA_sunF.cfg, see "PRIMECLUSTER Installation and Administration Guide."
If ALOM is properly configured.
For the special notes to configure the shutdown facility, see "PRIMECLUSTER Installation and Administration Guide."
For how to configure the ALOM and how to check the configuration, see "Advanced Lights Out Management (ALOM) CMT guide."
If the normal lamp of the port which is connected by HUBs and LAN cables is on.
If LAN cables are properly connected to the network control port (NET MGT) connector of ALOM or the HUB connector.
If ALOM has not restarted or upgraded the firmware.

# /opt/SMAW/bin/sdtool -e

# /opt/SMAW/bin/sdtool -b

When the connection fails again even after the above action is taken, hardware failure such as a network failure or a failure of ALOM or HUB may be the cause of the error. In this case, contact field engineers.

SA SA_vmchkhost to test host nodename failed
SA SA_vmgp to test host nodename failed
SA SA_vmSPgp to test host nodename failed
SA SA_vmSPgr to test host nodename failed

Content:

The node on which the message is displayed failed to check the connection to the host OS on the node nodename.

Corrective action:

Check the following:

If events such as panic or hang-up has not occurred on the host OS.
If the guest OS, host OS, or the network has high load.
If this message is no longer displayed after 10 minutes, the error may have been solved. Execute the following command to check if the shutdown facility works properly.
```
# /opt/SMAW/bin/sdtool -s
```
Check if the following configurations are correct when the output message is either SA_vmchkhost, SA_vmSPgp, or SA_vmSPgr.
- /etc/opt/SMAW/SMAWsf/SA_vmchkhost.cfg
- /etc/opt/SMAW/SMAWsf/SA_vmSPgp.cfg
- /etc/opt/SMAW/SMAWsf/SA_vmSPgr.cfg
Specify the account of the host OS and the login password of FJSVvmSP that are encrypted by the /opt/SMAW/bin/sfcipher command.
See "PRIMECLUSTER Installation and Administration Guide" to specify these files.
Check if the host OS information, which are the IP address, user name, and the node name of the host OS, are correct when the output message is SA_vmgp.
Use clrccusetup(1M) to check the configured console information. If there is any error in the information, use clrccusetup(1M) to register the console information again.
If a passphrase is not configured to connect to the hypervisor.
If the virtual IP addresses that are allocated for guest OSes are valid.
If the host OS or the guest OSes are configured properly.
See "PRIMECLUSTER Installation and Administration Guide" for configuration of the host OS or the guest OSes.
Check if the login user account for the shutdown facility is used to connect the guest OS (node) with the host OS by SSH, and the user authentication (such as creating RSA key) has been completed to connect by SSH for the first time.
Check If a password authentication is used as a user authentication to connect the guest OS (node) with the host OS by SSH. If an auto password authentication such as a public key authentication is used for the host OS, disable it.
If the normal lamp of the port which is connected by HUBs and LAN cables is on.
If LAN cables are properly connected to the PRIMEQUEST connector.

# /opt/SMAW/bin/sdtool -e

# /opt/SMAW/bin/sdtool -b

SA SA_vwvmr to test host nodename failed

Content:

The node on which the message is displayed failed to check the connection to vCenter Server.

Corrective action:

Check the following:

Make sure that the system or the network has high load.
If this message is no longer displayed after 10 minutes, the error may have been solved. Execute the following command to check if the shutdown facility works properly.
```
# /opt/SMAW/bin/sdtool -s
```
Make sure that the setting items of vCenter Server including IP address, port number, user name, and password specified in /etc/opt/SMAW/SMAWsf/SA_vwvmr.cfg are valid.
Make sure that the VM name specified in /etc/opt/SMAW/SMAWsf/SA_vwvmr.cfg is valid.
Make sure that /etc/opt/SMAW/SMAWsf/SA_vwvmr.cfg contains Japanese characters, and the character code of SA_vwvmr.cfg must be UTF-8.
For details on specifying /etc/opt/SMAW/SMAWsf/SA_vwvmr.cfg, see "PRIMECLUSTER Installation and Administration Guide."
Make sure that the necessary permission or the role to power off VM is set to the user specified in /etc/opt/SMAW/SMAWsf/SA_vwvmr.cfg.
For details on user permissions, see "PRIMECLUSTER Installation and Administration Guide."
Make sure that the network interface that are allocated for guest OSes are valid.
Make sure that the normal lamp of the port which is connected by HUBs and LAN cables is on.
Make sure that the cables are properly connected.

# /opt/SMAW/bin/sdtool -e
# /opt/SMAW/bin/sdtool -b

If the connection can still not be established after checking above items, the possible cause may be a hardware damage (e.g. network or hub failure).In that case, please contact field engineers.

When this corrective action does not work, record this message and collect information for an investigation. Then, contact field engineers. For details on how to collect information, see "Troubleshooting" in "PRIMECLUSTER Installation and Administration Guide."

SA SA_xscfsnmpg0p.so to test host nodename failed
SA SA_xscfsnmpg1p.so to test host nodename failed
SA SA_xscfsnmpg0r.so to test host nodename failed
SA SA_xscfsnmpg1r.so to test host nodename failed
SA SA_xscfsnmp0r.so to test host nodename failed
SA SA_xscfsnmp1r.so to test host nodename failed

Content:

The node on which the message is displayed failed to check the connection to XSCF on the node nodename.

Corrective action:

Check the following:

<Corrective action 1>

If the operation has been performed for the cluster after Migration.
If the guest domain, which was stopped by Cold Migration, has been started.

When either of the above is found to be the cause of the error, see "PRIMECLUSTER Installation and Administration Guide" to perform the operation after Migration.

<Corrective action 2>

If the configuration information of the logical domains has been saved.
If the states of the logical domains can be checked in XSCF.

When either of the above is found to be the cause of the error, use the ldm add-spconfig command to save the confirmation information of the logical domains.

<Corrective action 3>

Whether a load is placed on the system or the network.
If the console information of XSCF such as the IP address, node name, or the connection protocols (telnet and SSH) is correct.
Use clrccusetup(1M) to check the configured SNMP asynchronous monitoring information. If there is any error in the information, use clrccusetup(1M) to register the console information again.
If XSCF is configured properly.
For the special notes to configure the shutdown facility, see "PRIMECLUSTER Installation and Administration Guide."
For how to configure XSCF and how to check the configuration, see "Fujitsu SPARC M12 and Fujitsu M10/SPARC M10 System Operation and Administration Guide."
When SSH is used to connect to XSCF.
- Check if the login user account for the shutdown facility is used to connect the cluster node with XSCF by SSH, and the user authentication (such as creating RSA key) has been completed to connect by SSH for the first time.
- Check if a password authentication is used as a user authentication to connect the cluster node with XSCF by SSH.
- If an auto password authentication such as a public key authentication is used for XSCF, disable it. For how to configure XSCF and how to check the configuration, see "Fujitsu SPARC M12 and Fujitsu M10/SPARC M10 System Operation and Administration Guide."
If the normal lamp of the port which is connected by HUBs and LAN cables is on.
If LAN cables are properly connected to the XSCF-LAN port connector of XSCF or the HUB connector.
If the XSCF shell port, which belongs to the telnet port, is not connected from out of a cluster.
Connect to the XSCF shell port through a serial port to check the connection status. For how to connect and how to check the connection, see "Fujitsu SPARC M12 and Fujitsu M10/SPARC M10 System Operation and Administration Guide."
If the IP address of XSCF belongs to the same segment as the administrative LAN or the asynchronous monitoring sub-LAN.
If the firmware in XSCF has not been restarted or updated, or if an event such as XSCF failover has not occurred.

# /opt/SMAW/bin/sdtool -e
# /etc/opt/FJSVcluster/bin/clsnmpmonctl stop
# /etc/opt/FJSVcluster/bin/clsnmpmonctl start
# /opt/SMAW/bin/sdtool -b

<Corrective action 4>

If the port number (No.9385) of the SNMP trap receiver daemon (snmptrapd) that is used in the shutdown facility is duplicated with the port number of the other product.

When the above is found to be the cause of the error, see "Changing Port Numbers for SNMP" in "PRIMECLUSTER Installation and Administration Guide."

<Corrective action 5>

If the status of the SMF service (svc:/milestone/fjsvcldevtrap) in the shutdown facility is online.
Execute the following command to check the status of the fjsvcldevtrap service.
```
# svcs svc:/milestone/fjsvcldevtrap
```

If the status is either disabled or maintenance, take the following actions.

# /opt/SMAW/bin/sdtool -e
# /etc/opt/FJSVcluster/bin/clsnmpmonctl stop
# /etc/opt/FJSVcluster/bin/clsnmpmonctl start
# /opt/SMAW/bin/sdtool -b

<Corrective action 6>

If the SNMP agent of XSCF works correctly.
Log in to XSCF and execute the showsnmp command. Check the execution result.
```
XSCF> showsnmp
```
[Checklist]
- "Agent Status: Enabled" is output
- "Enabled MIB Modules: None" is not output

If the execution result does not match the expected output results in the checklist above, execute the following commands on XSCF.

XSCF> setsnmp disable
XSCF> setsnmp enable

After that, take the following actions on all cluster nodes.

# /opt/SMAW/bin/sdtool -e
# /opt/SMAW/bin/sdtool -b

<Other corrective actions>

If the error cause was not any one of the above check items, a network fault or a hardware fault such as XSCF or HUB may be the error cause. In this case, contact field engineers.

If the problem is not resolved by the above action, record this message and collect information for an investigation. Then, contact field engineers. For details on how to collect information, see "Troubleshooting" in "PRIMECLUSTER Installation and Administration Guide."

SA Shutdown Agent to init host nodename failed

Content:

Initializing the shutdown agent Monitoring Agent for the node <nodename> failed.

Corrective action:

Check if the shutdown agent is properly configured. For how to configure the shutdown agent, see "PRIMECLUSTER Installation and Administration Guide." If the configuration is correct, record this message and collect information for an investigation. Then, contact field engineers. For details on how to collect information, see "Troubleshooting" in "PRIMECLUSTER Installation and Administration Guide."

SA Shutdown Agent to test host nodename failed

Content:

The node on which the message is displayed failed to check the connection to the node nodename.

The connection destination varies depending on each shutdown agent.

Corrective action:

Check the following. The connection destination varies depending on each shutdown agent.

If the system or the network has high load.
If this message is no longer displayed after 10 minutes, the error may have been solved. Execute the following command to check if the shutdown facility works properly.
```
# /opt/SMAW/bin/sdtool -s
```
If the configuration of the shutdown daemon (/etc/opt/SMAW/SMAWsf/rcsd.cfg) or the configuration of the shutdown agent is correct.
If the configuration of the connection destination is correct.
When SSH is used to connect to the connection destination, if the login user account for the shutdown facility is used to connect the cluster node with the connection destination by SSH, and the user authentication (such as creating RSA key) has been completed to connect by SSH for the first time.
Check if a password authentication is used as a user authentication to connect the cluster node with the connection destination by SSH. If an auto password authentication such as a public key authentication is used for the connection destination, disable it.
If the cables are properly connected to the connection destination.

# /opt/SMAW/bin/sdtool -e

# /opt/SMAW/bin/sdtool -b

When the connection fails again even after the above action is taken, hardware failure such as a network failure, a failure of the connection destination or HUB may be the cause of the error. In this case, contact field engineers.

SA Shutdown Agent to unInit host nodename failed

Content:

The shutdown process of the shutdown agent <Shutdown Agent> for the node <nodename> failed.

Corrective action:

Collect information for an investigation. Then, contact field engineers. For details on how to collect information, see "Troubleshooting" in "PRIMECLUSTER Installation and Administration Guide."

select of CLI Pipe & RCSDNetPipe failed, errno errno

Content:

The select function in the shutdown daemon returns abnormally.

Corrective action:

The shutdown facility starts the operation to restart the pipe of CLI. No action is required because this does not affect other processes that are operated by the system.

string in file file around line number

Content:

The syntax in rcsd.cfg is incorrect.

Corrective action:

Correct the syntax error.

The attempted shutdown of cluster host nodename has failed

Content:

An internal error occurred in the program.

Corrective action:

The SF-CF event processing failed string, status value

Content:

When the "status 6147" is displayed during the shutdown processing, this means that the shutdown notification is received from other nodes.

When other messages are displayed, this means that an internal error occurred in the program.

Corrective action:

No corrective action is required when the "status 6147" is displayed during the shutdown processing. When the status is other than above, record this message and collect information for an investigation. Then, contact field engineers. For details on how to collect information, see "Troubleshooting" in "PRIMECLUSTER Installation and Administration Guide."

The SF-CF has failed to locate host nodename

Content:

The node name in the rcsd.cfg is not the CF node name.

Corrective action:

Specify the CF node name for the node name in the rcsd.cfg. Check the CF node name by cftool -n command. Correct the rcsd.cfg and execute the following command to restart the shutdown facility.

# /opt/SMAW/bin/sdtool -e

# /opt/SMAW/bin/sdtool -b

If the above action does not work, record this message and collect information for an investigation. Then, contact field engineers.

The SF-CF initialization failed, status value

Content:

CF is not configured or CF may not be loaded.

Corrective action:

Configure CF.
See the manual below for configuration.

"Examples for setting cluster" in "PRIMECLUSTER Cluster Foundation (CF) Configuration and Administration Guide."

The specified guest domain cannot be connected. (nodename:nodename)

Content:

The specified guest domain nodename cannot be connected.

Corrective action:

Check the following points:

Whether a load is placed on the system or the network.
If it is a temporary load, no corrective action is required.
Whether the configured guest OS information is correct.
- For Linux 4.3A30 or later
  Check whether the guest OS information set in /etc/opt/FJSVcluster/etc/kvmguests.conf is correct.
  If the guest OS information is incorrect, set the information in the kvmguests.conf file again.
- For Solaris 4.3A20 or later
  Execute the clovmmigratesetup -l command to check whether the guest OS information is correct.
  If the guest OS information is incorrect, set the correct information again.
  If the guest OS information is output when executing the clovmmigratesetup command, check whether the content is correct.
Whether the specified guest OS can be connected.
If the guest OS cannot be connected, review the network settings.
Check whether the general user privileges of the shutdown facility is being registered in the guest OS.
Whether the guest OS is connected from the host OS via SSH connection with the root user or general user privileges of the shutdown facility, and also the user inquiry of the SSH connection first time such as RSA Key Generation is complete.
Complete the user inquiry of the SSH connection first time such as RSA Key Generation if it is not complete.

If this corrective action does not work, write down the error message, collect required information for troubleshooting and contact field engineers. For details on how to collect information, see "Troubleshooting" in "PRIMECLUSTER Installation and Administration Guide."