4.2.12 Switchover Using the Batch Command [Switchover Destination Site]

Perform switchover using a batch command.

When the automatic collection of switchover information is enabled, performing switchover using the batch command disables the setting.

If the maintenance mode of image is set, release the maintenance mode of image before executing the batch command.

[Windows Manager]

>Installation_folder\SVROR\Manager\bin\rcxadm imagemgr set -attr vm.clone.maintenance=false <RETURN>

[Linux Manager]

# /opt/FJSVrcvmr/bin/rcxadm imagemgr set -attr vm.clone.maintenance=false <RETURN>

For Site Switchover
Execute the batch command on the switchover destination site to switch the site.
[Windows Manager]
>Installation_folder\SVROR\Manager\bin\rcxrecovery <RETURN>
[Linux Manager]
# /opt/FJSVrcvmr/bin/rcxrecovery <RETURN>

For Tenant Switchover

Execute the batch command on the switchover destination site to switch over tenants.

Note

When performing the switchover of the last tenant that will remain on the switchover source site, specify the -global option in addition to the -tenant option. For the Active-Active operation, do not use the -global option.

For Tenant Switchover (tenant1) in the Active-Active Operation
[Windows Manager]
>Installation_folder\SVROR\Manager\bin\rcxrecovery -tenant tenant1 <RETURN>
[Linux Manager]
# /opt/FJSVrcvmr/bin/rcxrecovery -tenant tenant1 <RETURN>
When switching over tenants (tenant2, tenant3) in the Active-Active operation (No tenants will remain on the switchover source site after the switchover operation)
[Windows Manager]
>Installation_folder\SVROR\Manager\bin\rcxrecovery -tenant tenant2,tenant3 -global <RETURN>
[Linux Manager]
# /opt/FJSVrcvmr/bin/rcxrecovery -tenant tenant2,tenant3 -global <RETURN>

For details on the rcxrecovery command, refer to "B.2 rcxrecovery".
The switchover status of the rcxrecovery command can be confirmed from output messages or the DR execution log.
For details on the DR execution log, refer to "D.2 Logs Regarding the rcxrecovery Command".

In addition, the following operations require manual operations or decisions by an operator. Respond to inquiries from the system to continue execution of the command.

For VMware virtual L-Server switchover (VM guest), the following messages are displayed.

It is necessary to manually delete the network settings on which cleanup is performed in the network settings "corresponding to network resources" configured on the switchover destination site beforehand. It is necessary to manually create the network (virtual switches, port groups) which has been configured on the switchover source site beforehand.
After the operation is complete, enter "y", and continue the command.

For network resources that have been cleaned up, clear their network configurations manually using VM management software or the network devices. Also, if you have manually configured the network on the primary site, configure the network for the backup site again.

Press [y] when the reconfiguration is complete. : y

When neither the -tenant option nor the -nocleanup option are specified for cleanup of the switchover destination site, an inquiry will be displayed to ask if you really want it to be cleaned up. Press "y" to approve the cleanup and continue the rcxrecovery command. If cleanup of the switchover destination site cannot be performed, press "n" to quit the command.
When VM management software is not configured on a physical L-Server using the physical/virtual L-Server switchover (VM host/VM guest) method, it is necessary to manually restore the VM management software during switchover. The rcxrecovery command will show a prompt while waiting for the restoration of VM management software. In that case, continue the command after restoration of VM management software is completed.
If there is a physical L-Server with VMware ESXi installed on the switchover target server, it is necessary to register the VM host with SVOM and register the Resource Orchestrator agent with the Resource Orchestrator manager during execution of the rcxrecovery command.
The command will be suspended until registration of the VM host and agent is completed. When the registration is completed, resume the command.
If the Resource Orchestrator agent has already been registered with the Resource Orchestrator manager, it is not necessary to register the agent again.

If the maintenance mode of images is released before execution of the batch command, set the maintenance mode of images as necessary.

[Windows Manager]

>Installation_folder\SVROR\Manager\bin\rcxadm imagemgr set -attr vm.clone.maintenance=true <RETURN>

[Linux Manager]

# /opt/FJSVrcvmr/bin/rcxadm imagemgr set -attr vm.clone.maintenance=true <RETURN>

Information

The rcxrecovery command performs the following:
- Mapping of storage
  Storage resource mapping is performed, based on the files defining the relations between the storage on the switchover source and destination sites.
- Division of import resources
  Imported resources are divided using the following units:
  - Resources in the global pool and resources belonging to individual tenants
  - Physical L-Server resources and virtual L-Server resources
- Deletion of the environment of the switchover destination site
  Deletes the following environments used for switchover destination site operations.
  - L-Platform Templates
  - L-Platform Configuration Information
  - Resource Information
  - Accounting Information
  - Metering Logs
  - Usage Charges
- Import Information
  Import the following information:
  - L-Platform Templates
  - L-Platform Configuration Information
  - Resource Information
  - Accounting Information
  - Metering Logs
  - Usage Charge Files
When performing Active-Active operation, the accounting information for the switchover source and destination sites is merged and then registered on the switchover destination site. Check the accounting information and modify or delete it if necessary.
For details on the procedures to modify and delete accounting information, refer to "15.3 Operate Accounting Information" in the "Operation Guide CE".

Note

Configured physical servers and linked L-Servers for which only configuration definition has been created are not covered by disaster recovery switchover.
To facilitate the operation for linking with L-Servers again after switchover, execute the following rcxadm config command in advance on the switchover source site, and then back up the configuration information.

[Windows Manager]

>Installation_folder\SVROR\Manager\bin\rcxadm config filter -convert -indir indir -outdir outdir <RETURN>

[Linux Manager]

# /opt/FJSVrcvmr/bin/rcxadm config filter -convert -indir indir -outdir outdir <RETURN>

The following information will be output to outdir, using the file name convert.txt.
After completing the restoration procedure, perform creation of a link with an L-Server and setting the scope again as shown below:

[command]
rcxadm lserver convert -with BX900_1 -name physical_lserver_1
rcxadm lserver convert -with BX900_3 -name physical_lserver_2 -to /tenant_1
rcxadm lserver convert -with BX900_6 -name physical_lserver_3 -label test_label3 -comment test_comment3 -to /folder_1
[user]
user_1, physical_lserver_1
user_1, /tenant_1/physical_lserver_2
user_2, /folder_1/physical_lserver_2
[usergroup]
usergroup_1, physical_lserver_1
usergroup_2, /folder_1/physical_lserver_3

Storage using the ETERNUS dynamic LUN mirroring is switched to the LUN that was prepared on the switchover destination site when the switchover operation is performed after a disaster. Operation may be different if the storage is not released even after deleting an L-Platform.
Perform operation so that the import of the resource configuration information and user definition file (XML file) using the rcxrecovery command is not executed at the same time as modification of the resource configuration information or user definition file (XML file) due to L-Server creation, or export operations using the rcxmgrexport command.
If the -allon option or the -alloff option is not specified for the rcxrecovery command, the power status of L-Servers recovered using the rcxrecovery command becomes the same as that when the switchover information was collected from the switchover source site. L-Servers start up in order of the priority set for L-Servers.
The RC console may not display the correct power status of L-Servers immediately after a disaster recovery switchover. Wait for about three minutes after the switchover to see the correct power status of L-Servers.
As the performance information (the information displayed in the dashboard and in operational status displays) depends on the configuration of the switchover source site, this is not carried over to the switchover destination site when switched over. The information on the switchover destination site is displayed.
When connection networks of ROR console on the admin server have different IP addresses between the switchover source and destination sites, and the FQDN is to be taken over by the switchover destination site, change the IP address for the FQDN registered in the DNS server to the IP address of the admin server on the switchover destination site after the switchover is performed.
When the FQDN is taken over from the switchover source site to the switchover destination site, clear the caches or temporary internet files of Web browsers that are used to connect to the ROR console on clients.
When test certificates are used, import the certificate of the switchover destination site to the browser if the certificate was not imported during installation.
For details, refer to "Chapter 5 Importing a Certificate to a Browser" in the "Setup Guide CE".
When switchover to the switchover destination site is performed, switchover may fail due to an import failure. In this case, take corrective action for the import failure and perform the switchover operation again.
If the failure occurred while the switchover process was being performed, performing switchover again will skip the process for the resources that have already been switched and only process the resources that have not been switched yet.
When switchover is performed with the setting that clears the configuration information of the destination site (without the -nocleanup option specified), if the process "cleanup of resources" reads "completed", perform the switchover again with the -nocleanup" option specified.
The following shows an example of the case where the VM guest could not be restored during L-Server import.
Example
Example 1
1. As an error occurred during import, the following error is displayed:
  >rcxrecovery <RETURN> FJSVrcx:ERROR:62569:/tenant1/l-platform1/l-server1:lserver:import was interrupted. Message=:67154: VM Guest not found
2. The status of the relevant resources is displayed to help determine the cause of the error.
  >rcxadm config show -type lserver -name /tenant1/l-platform1/l-server1 -dir dir1 -format xml<RETURN>
  For -dir, specify the folder that was most recently exported in the folder below.
  - The folder specified in the rcxrecovery -dir option
  [Windows Manager]
  - The import folder specified in #rcxrecovery_define of fa_dr.rcxprop\ManagerExport\ROR export_date
  [Linux Manager]
  - The import folder specified in #rcxrecovery_define of fa_dr.rcxprop/ManagerExport/ROR export_date
3. Delete the resource where the problem occurred from the export file. Make a note of the name of the deleted resource.
  >rcxadm config filter -exclude lserver -name /tenant1/l-platform1/l-server1 -indir dir1 -outdir dir2 <RETURN>
4. Use the export file output by the command in step 3 to overwrite the file in the folder specified for -indir in step 3.
5. Import again.
  >rcxrecovery -nocleanup<RETURN>
Example 2
1. As an error occurred during import, the following error is displayed:
  >rcxrecovery <RETURN> FJSVrcx:ERROR:67335:registering of VM guest failed. detail=(registervm L-ServerB-2:task result error, vmerrno=104,ip=192.168.201.70,host=192.168.201.115)
2. Identify the L-Server on which the error occurred.
  In the switchover information folder, check the l_servers.xml file that was most recently exported.
  [Windows Manager]
  The import folder specified in #rcxrecovery_define of fa_dr.rcxprop/ManagerExport/ROR_export_date\l_servers.xml
  [Linux Manager]
  The import folder specified in #rcxrecovery_define of fa_dr.rcxprop/ManagerExport/ROR_export_date/l_servers.xml
  In the xml file, find the VM guest name (L-ServerB-2) displayed in the error message, and check the name and id in the L-Server element in which the entry is included.
  <LServer name="L-ServerB" id="******">
  ...
  <VmGuest name="L-ServerB-2">
3. Check resource_folders.xml in the switchover information folder.
  When there are L-Platforms and tenants in which the L-Server with id checked in step 2 is included in the xml file, check the names.
  <Folder name="TenantB" tenant="TENANT">
  ...
  <Folder name="L-platformB" tenant="LPlatform">
  ...
  <LServer name="L-ServerB" id="******">
4. Delete the resource where the problem occurred from the export file. Make a note of the name of the deleted resource.
  >rcxadm config filter -exclude lserver -name /tenantB/l-platformB/L-ServerB -indir dir1 -outdir dir2 <RETURN>
  Add the tenant name, the L-Platform name, and the L-Server name checked in steps 2 and 3 to -name, connecting them with slashes ("/").
  - When a tenant is included
    /tenantB/l-platformB/L-Server-B
  - When a tenant is not included
    /L-Server-B
  Specify the folder checked in step 2 for -indir.
  [Windows Manager]
  The import folder specified in #rcxrecovery_define of fa_dr.rcxprop/ManagerExport/ROR_export_date
  [Linux Manager]
  The import folder specified in #rcxrecovery_define of fa_dr.rcxprop/ManagerExport/ROR_export_date
5. Use the export file output by the command in step 4 to overwrite the file in the folder specified for -indir in step 4.
6. Import again.
  >rcxrecovery -nocleanup<RETURN>

Point

When the automatic collection of switchover information is enabled, performing cleanup disables the setting.
Refer to "4.2.16 Exporting Configuration Information and Resuming Automatic Collection of Switchover Information [Switchover Destination Site]" to enable the setting when necessary.