PRIMECLUSTER Messages

5.2.2 Information and Warning Messages for which Corrective Action is Necessary

Advertisement to host:nodename on admIP:string failed

Content:

When the nodes are forcibly stopped, the data of forcible shutdown request may not be sent successfully to the node. When this message is output, an unintended node may be forcibly stopped when the nodes are forcibly stopped.

Corrective action:

Record this message and collect information for an investigation. Then, contact field engineers. For details on how to collect information, see "Troubleshooting" in "PRIMECLUSTER Installation and Administration Guide."

All cluster hosts have NOT reported their weight

Content:

When the nodes are forcibly stopped, the data was not successfully received from all cluster nodes. When this message is output, an unintended node may be forcibly stopped when the nodes are forcibly stopped.

Corrective action:

Record this message and collect information for an investigation. Then, contact field engineers. For details on how to collect information, see "Troubleshooting" in "PRIMECLUSTER Installation and Administration Guide."

A request to exit rcsd came in during a shutdown cycle, this request was ignored

Content:

The rcsd daemon (sdtool -e) was to be stopped when the rcsd was trying to stop the nodes.

Corrective action:

Retry after the shutdown task is completed.

Cannot open CIP configuration file : file

Content:

A network communication by the shutdown daemon is impossible because the configuration definition file of CIP was not opened successfully. When this message is output, an unintended node may be forcibly stopped when the nodes are forcibly stopped.

Corrective action:

Search the configuration definition file of CIP.

If the file does not exist, configure CIP. See "PRIMECLUSTER Installation and Administration Guide" for configuration.

If the file exists, execute the following commands to restart the shutdown facility.

# /opt/SMAW/bin/sdtool -e

# /opt/SMAW/bin/sdtool -b

When this corrective action does not work, record this message and collect information for an investigation. Then, contact field engineers. For details on how to collect information, see "Troubleshooting" in "PRIMECLUSTER Installation and Administration Guide."

checkAdmInterface : can't open datagram socket. errno=errno

Content:

A network communication by the shutdown daemon is impossible. When this message is output, an unintended node may be forcibly stopped when the nodes are forcibly stopped.

Corrective action:

Execute the following commands to restart the shutdown facility.

# /opt/SMAW/bin/sdtool -e

# /opt/SMAW/bin/sdtool -b

When this corrective action does not work, record this message and collect information for an investigation. Then, contact field engineers. For details on how to collect information, see "Troubleshooting" in "PRIMECLUSTER Installation and Administration Guide."

cleanUpServerThread: Failed to cancel advertisement server thread

Content:

A communication network thread for the shutdown daemon was not recovered successfully. When this message is output, an unintended node may be forcibly stopped when the nodes are forcibly stopped.

Corrective action:

Execute the following commands to restart the shutdown facility.

# /opt/SMAW/bin/sdtool -e

# /opt/SMAW/bin/sdtool -b

Execute the following command and check the result.

# /opt/SMAW/bin/sdtool -s

• When Init State is InitWorked and Test State is TestWorked

  No action is required.

• When Init State is InitFailed or Test State is TestFailed

  Record this message and collect information for an investigation. Then, contact field engineers. For details on how to collect information, see "Troubleshooting" in "PRIMECLUSTER Installation and Administration Guide."

• Other than the above status

  Wait for a while before executing the above commands again. Check the result.

command timed out after 0.1 sec

Content:

The command times out. It was not executed correctly.

When this message is output, an unintended node may be forcibly stopped when the nodes are forcibly stopped.

Corrective action:

Record this message and collect information for an investigation. Then, contact field engineers. For details on how to collect information, see "Troubleshooting" in "PRIMECLUSTER Installation and Administration Guide."

Failed in plock(). errno errno

Content:

The memory in the shutdown daemon process was not locked successfully. When the system load is high, an unintended node may be forcibly stopped when the nodes are forcibly stopped.

Corrective action:

Execute the following commands to restart the shutdown facility (SF).

# /opt/SMAW/bin/sdtool -e

# /opt/SMAW/bin/sdtool -b

If the conditions are satisfied, record this message and collect information for an investigation. Then, contact field engineers. For details on how to collect information, see "Troubleshooting" in "PRIMECLUSTER Installation and Administration Guide."

Failed in priocntl(option). errno errno. RCSD is not a real-time process

Content:

Execution priority of the shutdown daemon remains low. When the system load is high, an unintended node may be forcibly stopped when the nodes are forcibly stopped.

Corrective action:

Execute the following commands to restart the shutdown facility (SF).

# /opt/SMAW/bin/sdtool -e

# /opt/SMAW/bin/sdtool -b

If the conditions are satisfied, record this message and collect information for an investigation. Then, contact field engineers. For details on how to collect information, see "Troubleshooting" in "PRIMECLUSTER Installation and Administration Guide."

Failed to cancel thread of string

Content:

A thread was not cancelled successfully.

Corrective action:

Record this message and collect information for an investigation. Then, contact field engineers. For details on how to collect information, see "Troubleshooting" in "PRIMECLUSTER Installation and Administration Guide."

Failed to do fcntl(serversockfd, FD_CLOEXEC) errno errno

Content:

An internal error occurred in the program.

Corrective action:

Record this message and collect information for an investigation. Then, contact field engineers. For details on how to collect information, see "Troubleshooting" in "PRIMECLUSTER Installation and Administration Guide."

Failed to do string, reason (value)string

Content:

An internal function was not called successfully.

Corrective action:

Record this message and collect information for an investigation. Then, contact field engineers. For details on how to collect information, see "Troubleshooting" in "PRIMECLUSTER Installation and Administration Guide."

Failed to get nodeid for host nodename. reason (value)string

Content:

A cluster node id of a node cannot be obtained.

Corrective action:

Record this message and collect information for an investigation. Then, contact field engineers. For details on how to collect information, see "Troubleshooting" in "PRIMECLUSTER Installation and Administration Guide."

Failed to open CLI response pipe for PID pid, errno errno

Content:

The pipe cannot be opened because the rcsd daemon responds to sdtool.

Corrective action:

Record this message and collect information for an investigation. Then, contact field engineers. For details on how to collect information, see "Troubleshooting" in "PRIMECLUSTER Installation and Administration Guide."

Failed to open lock file

Content:

An internal error occurred in the program.

Corrective action:

Record this message and collect information for an investigation. Then, contact field engineers. For details on how to collect information, see "Troubleshooting" in "PRIMECLUSTER Installation and Administration Guide."

Failed to perform delay

Content:

Forcible shutdown process of the nodes was suspended because an error occurred during the process. When this message is output, the unintended node may be forcibly stopped when the node is forcibly stopped.

However, if the node is forcibly stopped successfully after the following steps 1 to 3, no corrective action is required.

1. Any one of the following operations is executed.

   • The node is forcibly stopped manually by executing the sdtool -k command.

   • The node is forcibly stopped due to the double failure of the userApplication.

   • The node is forcibly stopped when the userApplication is forcibly operated by the operator intervention message.

2. The node is forcibly stopped successfully after step 1.

3. This message is output to the forcibly stopped node after step 2.

Corrective action:

Record this message and collect information for an investigation. Then, contact field engineers. For details on how to collect information, see "Troubleshooting" in "PRIMECLUSTER Installation and Administration Guide."

Failed to read the received advertisement from the rcsd net pipe

Content:

The forcible shutdown request from other nodes may not be received successfully. When this message is output, an unintended node may be forcibly stopped when the nodes are forcibly stopped.

Corrective action:

Record this message and collect information for an investigation. Then, contact field engineers. For details on how to collect information, see "Troubleshooting" in "PRIMECLUSTER Installation and Administration Guide."

host information for string not found
gethostbyname returned Invalid address for string

Content:

A network communication of the shutdown daemon is impossible because the system has problems inside, or the invalid host name or the invalid IP address is specified either in the configuration definition file of CIP or the configuration definition file of the shutdown daemon. When this message is output, an unintended node may be forcibly stopped when the nodes are forcibly stopped.

Corrective action:

Check if the host name or the IP address, which is specified either in the configuration definition file of CIP or the configuration definition file of the shutdown daemon, is correct.

If the invalid host name or IP address is specified, configure CIP or the shutdown facility properly. See "PRIMECLUSTER Installation and Administration Guide" for configuration.

# /opt/SMAW/bin/sdtool -e

# /opt/SMAW/bin/sdtool -b

If the conditions are satisfied, record this message and collect information for an investigation. Then, contact field engineers. For details on how to collect information, see "Troubleshooting" in "PRIMECLUSTER Installation and Administration Guide."

host nodename has no input in 2 seconds. Ignore it

Content:

No data was sent from the node nodename during the queuing time for synchronization when the nodes were forcibly stopped. When this message is output, an unintended node may be forcibly stopped when the nodes are forcibly stopped.

Corrective action:

Record this message and collect information for an investigation. Then, contact field engineers. For details on how to collect information, see "Troubleshooting" in "PRIMECLUSTER Installation and Administration Guide."

Host nodename, MA Monitoring Agent, MAHostGetState() failed

Content:

MAHostGetState() returned an abnormal value when MAHostGetState() was executed in the MA monitor thread.

Corrective action:

Record this message and collect information for an investigation. Then, contact field engineers. For details on how to collect information, see "Troubleshooting" in "PRIMECLUSTER Installation and Administration Guide."

Local host is not defined in rcsd.cfg

Content:

A CF node name of the node that this message is output is not specified for the rcsd cfg. When this message is output, an unintended node may be forcibly stopped when the nodes are forcibly stopped.

Corrective action:

Check if the CF node name of the cluster node that this message is output is specified for the rcsd cfg. Check the CF node name by cftool -n command.

If the CF node name is invalid, correct the rcsd.cfg

Execute the following commands to restart the shutdown facility.

# /opt/SMAW/bin/sdtool -e

# /opt/SMAW/bin/sdtool -b

If the conditions are satisfied, record this message and collect information for an investigation. Then, contact field engineers. For details on how to collect information, see "Troubleshooting" in "PRIMECLUSTER Installation and Administration Guide."

makeXDRfromAdv: can't convert NULL ad to XDR

Content:

An internal error occurred in the shutdown facility.

Corrective action:

Record this message and collect information for an investigation. Then, contact field engineers. For details on how to collect information, see "Troubleshooting" in "PRIMECLUSTER Installation and Administration Guide."

No/Invalid admin LAN specified. Advertisement server will not be started

Content:

The network communication thread of the shutdown daemon was not created successfully because the invalid IP address or the invalid host name is specified in the rcsd.cfg. When this message is output, an unintended node may be forcibly stopped when the nodes are forcibly stopped.

Corrective action:

Review the IP address or the host name that is specified in the rcsd.cfg and then execute the following commands to restart the shutdown facility.

# /opt/SMAW/bin/sdtool -e

# /opt/SMAW/bin/sdtool -b

If the conditions are satisfied, record this message and collect information for an investigation. Then, contact field engineers. For details on how to collect information, see "Troubleshooting" in "PRIMECLUSTER Installation and Administration Guide."

open failed on rcsd net pipe name, errno errno

Content:

Data from other nodes may not be received successfully. When this message is output, an unintended node may be forcibly stopped when the nodes are forcibly stopped.

Corrective action:

Record this message and collect information for an investigation. Then, contact field engineers. For details on how to collect information, see "Troubleshooting" in "PRIMECLUSTER Installation and Administration Guide."

open failed on RCSD response pipe name, errno errno

Content:

The pipe for rcsd could not be opened.

Corrective action:

Record this message and collect information for an investigation. Then, contact field engineers. For details on how to collect information, see "Troubleshooting" in "PRIMECLUSTER Installation and Administration Guide."

PID pid exitted due to receiving signal number number

Content:

The shutdown agent returned abnormally.

Corrective action:

Check the messages of the shutdown agent that will be output after this message is output for the corrective action. If no additional messages were output, record this message and collect information for an investigation. Then, contact field engineers. For details on how to collect information, see "Troubleshooting" in "PRIMECLUSTER Installation and Administration Guide."

PID pid exitted with a non-zero value of value

Content:

The shutdown agent returned abnormally.

Corrective action:

Check the messages of the shutdown agent that will be output after this message is output for the corrective action.

If no additional messages were output, record this message and collect information for an investigation. Then, contact field engineers. For details on how to collect information, see "Troubleshooting" in "PRIMECLUSTER Installation and Administration Guide."

PID pid was stopped with signal number number

Content:

The shutdown agent returned abnormally.

Corrective action:

Check the messages of the shutdown agent that will be output after this message is output for the corrective action.

If no additional messages were output, record this message and collect information for an investigation. Then, contact field engineers. For details on how to collect information, see "Troubleshooting" in "PRIMECLUSTER Installation and Administration Guide."

Pthread failed: pthread_XXXX : errcode num string

Content:

Library function pthread_XXXX was not executed successfully.

Corrective action:

Record this message and collect information for an investigation. Then, contact field engineers. For details on how to collect information, see "Troubleshooting" in "PRIMECLUSTER Installation and Administration Guide."

WARNING : Pid process id is not able to be terminated. The SA Shutdown Agent is now disabled from host  nodename

Content:

The shutdown agent Shutdown Agent for the node nodename could not be used.

Corrective action:

If the following message is displayed after this message, no action is required because the shutdown facility has already been restarted.

rcsd died abnormally. Restart it.

If the message is not displayed, execute the following commands to restart the shutdown facility.

# /opt/SMAW/bin/sdtool -e

# /opt/SMAW/bin/sdtool -b

If the conditions are satisfied, record this message and collect information for an investigation. Then, contact field engineers. For details on how to collect information, see "Troubleshooting" in "PRIMECLUSTER Installation and Administration Guide."

popen failed for command.errno = errno

Content:

The command command was not started successfully.

When this message is output, an unintended node may be forcibly stopped when the nodes are forcibly stopped.

Corrective action:

Record this message and collect information for an investigation. Then, contact field engineers. For details on how to collect information, see "Troubleshooting" in "PRIMECLUSTER Installation and Administration Guide."

SA Shutdown Agent to shutdown host nodename failed

Content:

The shutdown agent Shutdown Agent failed to forcibly stop the node nodename.

Corrective action:

Check the cluster node or the cluster application status by Cluster Admin. Start the cluster node or switch the cluster application if needed.

Record this message and collect information for an investigation. Then, contact field engineers. For details on how to collect information, see "Troubleshooting" in "PRIMECLUSTER Installation and Administration Guide."

Note

This message may be output if error occurs in the ESXi host in an environment that uses VMware vSphere HA. However, if the virtual machine has successfully migrated by VMware vSphere HA and the operation has resumed, no action is required.

SA string does not exist

Content:

The specified shutdown agent does not exist.

Corrective action:

Check the shutdown agent name that is specified in the rcsd.cfg. See "PRIMECLUSTER Installation and Administration Guide" for a shutdown agent name.

Execute the following commands to restart the shutdown facility.

# /opt/SMAW/bin/sdtool -e

# /opt/SMAW/bin/sdtool -b

If the conditions are satisfied, record this message and collect information for an investigation. Then, contact field engineers. For details on how to collect information, see "Troubleshooting" in "PRIMECLUSTER Installation and Administration Guide."

Sending type to host nodename failed, ackId=number

Content:

The data may not be sent to the node nodename successfully when the nodes were forcibly stopped. When this message is output, an unintended node may be forcibly stopped when the nodes are forcibly stopped.

Corrective action:

Record this message and collect information for an investigation. Then, contact field engineers. For details on how to collect information, see "Troubleshooting" in "PRIMECLUSTER Installation and Administration Guide."

Shutdown Agent Shutdown Agent timeout for host <nodename> is less than 20 seconds

Content:

The 'timeout' seconds of the shutdown agent of the node, which is specified in the rcsd.cfg, are less than 20 seconds.

Corrective action:

Check the 'timeout' seconds of the shutdown agent of the node, which is specified in the rcsd.cfg. See "PRIMECLUSTER Installation and Administration Guide" for the 'timeout' setting.

When the 'timeout' is corrected, execute the following commands to restart the shutdown facility.

# /opt/SMAW/bin/sdtool -e

# /opt/SMAW/bin/sdtool -b

The RCSD on host nodename is NOT running in CF mode

Content:

CF has not been installed in the node nodename.

Corrective action:

The PRIMECLUSTER package may not been installed in the system properly. Check that no error occurred when the package was installed in the system.

If the package is installed properly, record this message and collect information for an investigation. Then, contact field engineers. For details on how to collect information, see "Troubleshooting" in "PRIMECLUSTER Installation and Administration Guide."

The SA Shutdown Agent to action host nodename has exceeded its configured timeout, pid process id will be terminated

Content:

When the action action is executed for the node nodename, the shutdown agent did not return within 'timeout' seconds, which is the time configured in the rcsd.cfg.

Corrective action:

Check that if 'timeout' seconds that is configured in the rcsd.cfg is valid. See "PRIMECLUSTER Installation and Administration Guide" for 'timeout' configuration.

When 'timeout' seconds are tuned, execute the following commands to restart the shutdown facility.

# /opt/SMAW/bin/sdtool -e

# /opt/SMAW/bin/sdtool -b

If the conditions are satisfied, record this message and collect information for an investigation. Then, contact field engineers. For details on how to collect information, see "Troubleshooting" in "PRIMECLUSTER Installation and Administration Guide."

The SF-CF failed to declare host nodename(nodeid number) string, reason (value)string

Content:

An internal error occurred in the program.

Corrective action:

Record this message and collect information for an investigation. Then, contact field engineers. For details on how to collect information, see "Troubleshooting" in "PRIMECLUSTER Installation and Administration Guide."

Unknown host nodename

Content:

Errors were detected in the nodes other than the cluster node. When this message is output, the cluster configuration may include errors.

Corrective action:

Record this message and collect information for an investigation. Then, contact field engineers. For details on how to collect information, see "Troubleshooting" in "PRIMECLUSTER Installation and Administration Guide."

WARNING: No context allocation. MA Monitoring Agent for host nodename is neglected

Content:

Initializing the shutdown agent Monitoring Agent failed.

Corrective action:

After this message is output, the shutdown facility retries to initialize the Monitoring Agent every two minutes until the initialization is successfully completed. No action is required when the Monitoring Agent is automatically recovered by this process. If the recovery failed and the message is output again, take the corrective action for another message which is output at the same time. If the recovery failed again after taking the corrective action, record this message and collect information for an investigation. Then, contact field engineers. For details on how to collect information, see "Troubleshooting" in "PRIMECLUSTER Installation and Administration Guide."

write failed on rcsd net pipe name, errno errno

Content:

Data from other nodes may not be received successfully. When this message is output, an unintended node may be forcibly stopped when the nodes are forcibly stopped.

Corrective action:

Record this message and collect information for an investigation. Then, contact field engineers. For details on how to collect information, see "Troubleshooting" in "PRIMECLUSTER Installation and Administration Guide."