Collecting Initial Troubleshooting Data

About Collection Methods

The initial troubleshooting data can be collected using the following methods.

Based on each collection method's characteristics, chose the appropriate one according to the environment and system state in which the problem occurred.

Collecting data from the admin server
This method involves executing a troubleshooting data collection command (rcxadm mgrctl snap -all) on the admin server.

This method allows collection of data from all managed servers at once via the network. This is easier than executing a command individually on each managed server.

For how to collect the data, refer to "Collecting Data from the Admin Server (rcxadm mgrctl snap -all)".
Executing the rcxadm mgrctl snap -all command requires approximately the free space determined by the following formula: Number_of_registered_servers * 30 MB + 65 MB
Collecting data separately from each server
This method involves executing a data collection command (rcxadm mgrctl snap or rcxadm agtctl snap) on each server.

For how to collect the data, refer to "Collecting Data Separately from each Server (rcxadm mgrctl snap, rcxadm agtctl snap)".

Around 65 MB of free space is required to execute the rcxadm mgrctl snap command,

and around 30 MB of free space is required to execute the rcxadm agtctl snap command.
Collecting HBA address rename Setup Service Environment Data
This method involves collecting data that is required for identifying the cause of a problem that occurs on the HBA address rename setup service.

For how to collect the data, refer to "Collecting HBA address rename Setup Service Environment Data".

About 8 MB of free space is required to collect troubleshooting data for the HBA address rename server.

Note

[VMware]

For VMware ESX 4, also collect and send the following data to Fujitsu technical staff.
- Use the copy command to collect and send the following files.
  /var/log/vmware/hostd.log
  
  /var/log/vmware/hostd-*.log
  
  /var/log/vmware/hostd-*.log.gz
Collect and send the result (output) of the following command.
/usr/sbin/esxcfg-firewall -q
For VMware ESXi, perform the following procedures on the server from which data is to be collected:
- When VMware ESXi is not registered with VM management software
  1. Connect with VMware ESXi on the target VMware client.
  2. Select [Export]-[Export System Logs] from the [File] menu.
    The [Export System Logs] dialog is displayed.
  3. Specify the system log download destination.
  4. Click the [OK] button.
- When VMware ESXi is registered with VM management software
  1. Connect with VM management software on the VMware client.
  2. Select [Export]-[Export System Logs] from the [File] menu.
    The [Export System Logs] dialog is displayed.
  3. Select the target VMware ESXi.
  4. Specify the system log download destination.
  5. Click the [OK] button.
  Send the data output to the download destination to Fujitsu technical staff.

For VMware ESXi 6.5 or later, perform one of the following procedures on the server from which data is to be collected.

When VMware ESXi is not registered with VM management software
1. Use VMware Host Client to connect to the target VMware ESXi.
2. Right-click [Host] in the navigator, and select [Generate support bundle] from the popup menu.
  A dialog is displayed once generation of the support bundle is complete.
3. Click the [Download] button.
  A download confirmation message from the Web browser is displayed.
4. Specify the download destination, and save the logs.
When VMware ESXi is registered with VM management software
1. Connect to VM management software using vSphere Web Client.
2. Select [Hosts and Clusters] in the Inventory.
3. Right-click the target VMware ESXi host, and select [Export System Logs] from the popup menu.
  The [Export System Logs] dialog is displayed.
4. Click the [Finish] button.
  The [Download system logs to the following location] dialog is displayed.
5. Specify the download destination, and save the logs.
Send the data output to the download destination to Fujitsu technical staff.

Note

There are cases where the following error message is output in the /var/log/vmware/vsphere-client/logs/vsphere_client_virgo.log file, and the downloaded log is invalid.

In such cases, there is a chance that it is a known VMware problem, so refer to the release notes of vCenter Server and take the appropriate action.

com.vmware.vsphere.client.logbundle.DownloadLogController

Error downloading logs. org.apache.catalina.connector.ClientAbortException:

java.net.SocketException: Broken pipe (Write failed)

at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:407)

[Windows]

For Windows, also collect the output result of command execution using the following procedure.

Collect and send the output result (applied patch list of UpdateAdvisor (middleware)) of the following command to Fujitsu technical staff.

UpdateAdvisor_installation_destination\uam showup

Collect and send the output result (information of firewall configuration) of the following command to Fujitsu technical staff.

When using Windows Server 2012 or a later version

netsh advfirewall show allprofile

netsh advfirewall firewall show rule name=all verbose

When the LAN Manager modules are stored in the following location, use the copy command to directly collect the LAN Manager modules, and then send them to Fujitsu technical staff.
- Manager_installation_destination\ScwPro\tftp\agent\dos\boot\NETBIND.COM
- Manager_installation_destination\ScwPro\tftp\agent\dos\boot\PROTMAN.DOS
- Manager_installation_destination\ScwPro\tftp\agent\dos\boot\PROTMAN.EXE

[Linux]

For Linux, also collect the output result of command execution using the following procedure.
- Collect and send the output result (table information of packet filter rules) of the following command.
  
  iptables --list --verbose

Collecting Data from the Admin Server (rcxadm mgrctl snap -all)

Troubleshooting data from every managed server can be collected all at once by executing the command for collecting troubleshooting data, rcxadm mgrctl snap -all, on an admin server.

The following method is used to collect data with the data collection command, rcxadm mgrctl snap -all.

Collection Method

Perform the following procedure on the admin server to collect investigation data:

Log on to the admin server with OS administrative privileges.
OS administrator privileges are required to collect troubleshooting data.

Execute the rcxadm mgrctl snap -all command.

[Windows Manager]

>"Installation_folder\SVROR\Manager\bin\rcxadm" mgrctl snap [-dir directory] -all

[Linux Manager]

# /opt/FJSVrcvmr/bin/rcxadm mgrctl snap [-dir directory] -all

Send the collected troubleshooting data to Fujitsu technical staff.

Note

The manager must be running on the admin server in order to collect data from the admin server. If the manager cannot be run, collect data from each server.
When collecting data via the admin server, data cannot be collected from a managed server in the following cases.
- Communication cannot be established with the server
- The managed server is stopped
In either case, however, data will continue to be collected from other managed servers.

Use the command execution logs to check the execution results.

For details, refer to "5.11 rcxadm mgrctl" in the "Reference Guide (Command) VE".

For managed servers where data collection has failed, collect data by either executing the rcxadm mgrctl snap -all command on the admin server again, or by executing the rcxadm agtctl snap command on the managed server where data collection failed.

[VMware]

When a managed server is VMware ESXi, data cannot be collected from that managed server.
Use the procedure described in "About Collection Methods" when collecting data.

Collecting Data Separately from each Server (rcxadm mgrctl snap, rcxadm agtctl snap)

In addition to the rcxadm mgrctl snap -all command, which can collect troubleshooting data from all the managed servers at once, the commands rcxadm mgrctl snap and rcxadm agtctl snap are also provided. These collect data from only the server on which they are executed.

Use the following procedure to collect data with the rcxadm mgrctl snap command or the rcxadm agtctl snap command.

Collection Method

Collect data on the server using the following method.

Log in to the server with OS administrator privileges.
OS administrator privileges are required to collect troubleshooting data.

Execute the rcxadm mgrctl snap command or the rcxadm agtctl snap command.

Note that the command differs depending on the server from which data is collected.

For Admin Servers:

[Windows Manager]

>"Installation_folder\SVROR\Manager\bin\rcxadm" mgrctl snap [-dir directory]

[Linux Manager]

# /opt/FJSVrcvmr/bin/rcxadm mgrctl snap [-dir directory]

For Managed Servers:

[Windows] [Hyper-V]

>"Installation_folder\Agent\bin\rcxadm" agtctl snap [-dir directory]

[Linux] [VMware] [KVM]

# /opt/FJSVrcxat/bin/rcxadm agtctl snap [-dir directory]

[Solaris] [Solaris Zones]

# /opt/FJSVrcvat/bin/rcxadm agtctl snap [-dir directory]

Send the collected troubleshooting data to Fujitsu technical staff.

For details, refer to "5.2 rcxadm agtctl" or "5.11 rcxadm mgrctl" in the "Reference Guide (Command) VE".

Note

[VMware]

When the managed server is VMware ESXi, collect data using the procedure described in "About Collection Methods".

Collecting HBA address rename Setup Service Environment Data

The following method is used to collect the troubleshooting data necessary to investigate problems that occur on an HBA address rename setup service.

Collection Method

Perform the following procedure on the HBA address rename server to collect troubleshooting data.

Collect files from the HBA address rename server.
[Windows]

From the HBA address rename server, collect the log file installation_folder\WWN Recovery\rcxinstaller.log, and all files and folders under the following folder: installation_folder\WWN Recovery\var.

[Linux]

Collect the log file /var/tmp/rcxhbainstaller.log, and all files and folders under the following folder: /var/opt/FJSVrcvhb.

However, depending on the settings of the OS, the /var/tmp/rcxhbainstaller.log file may be deleted at some point after installation is complete. If trouble happens during installation, promptly collect the log file.
Obtain OS system logs.
[Windows]

Collect event logs (application and system logs).

[Linux]

Collect all /var/log/messages and /var/log/messages.X files. (The messages.X part represents the file name such as messages.1 or messages.2)
Take a screenshot of the window displayed when the problem occurred.
Perform the following operation on the HBA address rename server from which the data is to be collected in order to take a screenshot of the window:

Press the [Print Screen] key to copy image data onto the Clipboard, and then paste the image data into an image editing tool and save it as a bitmap file.

Obtain system version information.

[Windows]

From Windows explorer, right-click the "My Computer" icon, and select [Properties] from the popup menu.

In the [System Properties] dialog click the [General] tab and use the procedure explained in step 3 to obtain information such as the OS type, version and level, and service pack number.

[Linux]

Execute and collect the results of the following commands.

# cat /etc/redhat-release

# cat /proc/version

# cat /proc/meminfo

# cat /proc/cpuinfo

# rpm -qa

Send the collected troubleshooting data to Fujitsu technical staff.

1.2.1 Collecting Initial Troubleshooting Data

About Collection Methods

Collecting Data from the Admin Server (rcxadm mgrctl snap -all)

Collection Method

Collecting Data Separately from each Server (rcxadm mgrctl snap, rcxadm agtctl snap)

Collection Method

Collecting HBA address rename Setup Service Environment Data

Collection Method