This section explains operations of disaster recovery switchover when a disaster strikes the primary site.
The same operations can be used for failback from the backup site to the recovered primary site.
For switchover operations during a disaster, there are the two main operations described below.
Operations to Perform Batch Switchover of All Resources
Restore global resources and all tenants.
For details, refer to "Detailed Procedure for Switchover Operation during Disasters".
Figure 4.1 Operations to Perform Batch Switchover of All Resources
Operation Involving Resource Switchover by Tenant
Restore each tenant, after restoring global resources.
Restore global resources
Figure 4.2 Restore Global Resources
Restore each tenant resource
Figure 4.3 Restore Each Tenant Resource
For details, refer to "Detailed Procedure for Switchover Operation during Disasters".
Detailed Procedure for Switchover Operation during Disasters
The details of the procedure are explained below.
Stop operation of the backup site
Stop the operations being performed on the backup site, and prepare to switch over from the primary site.
Disconnect the storage
Stop the storage replication function, and disconnect the backup site from the primary site.
Mount volumes which contain the information for failover
Mount the volume which was replicated from the primary site when performing "2. Configure the Disaster Recovery function" in "Chapter 3 Installation", to the manager of the backup site.
Copy physical L-Server images
Physical L-Server images for the primary site are replicated to the backup site. However, the image file storage folder of the backup site contains the physical L-Server images for the backup site. Perform the following operation to switch the physical L-Server images on the backup site.
Save the physical L-Server images in the image file storage folder on the backup site to a different folder.
The image file storage folder can be shown by executing the rcxadm imagemgr command with the info option.
For details on rcxadm imagemgr command, refer to "5.9 rcxadm imagemgr" in the "Reference Guide (Command/XML) CE".
Clear the content of the image file storage folder.
Copy the physical L-Server images for the primary site to the image file storage folder.
The physical L-Server images for the primary site can be found in the Image folder of the volume mounted in "3. Mount volumes which contain the information for failover".
Restart the manager.
Copy firewall configurations
When using the firewall auto-configuration function for network devices, copy the newest configuration from the primary site to the firewall of the backup site.
Recover customized information of the primary site
If any files in the following folder on the primary site have been modified, restore the files from the backup saved in "2. Backup customize information of the primary site" in "4.1 Normal Operation", and restart manager if necessary.
[Windows Manager]
The following files under Installation_folder\SVROR\Manager\etc\customize_data
l_server.rcxprop
logctl_env.rcxprop
storage_rcxprop
vnetwork_vmware.rcxprop
vnetwork_hyperv.rcxprop
vnetwork_excluded_vmware.rcxprop
rcx_base.rcxprof
server_spec.rcxprop
library_share_deny.conf
image_admin_hyperv.rcxprop
library_share_User_Group_Name_deny.conf
kb978336_scvmm.rcxprop
image_admin_hyperv_User_Group_Name.rcxprop
server_control.rcxprop
vm.rcxprop
vm_console.rcxprop
scvmm_mac_pool.rcxprop
scvmm_mac_pool_Tenant_Name.rcxprop
vm_VMware.rcxprop
vm_Hyper-V.rcxprop
create_image_previous_folder_name_scvmm.rcxprop
auto_replace.rcxprop
Physical.rcxprop
spare_server_config.rcxprop
Installation_folder\SVROR\Manager\etc\trapop.bat
All files under Installation_folder\SVROR\Manager\etc\event_handler
All files under Installation_folder\SVROR\Manager\etc\scripts
The following file under Installation_folder\SVROR\Manager\etc\vm
vds_vc
Reflecting changes of settings of the primary site
Reflect the following customization settings of the primary site, which are performed according to "Chapter 19 To Customize Environment" in the "Setup Guide CE", on the backup site.
"19.1 Settings for Sending Email"
"19.2.3 Setting Application process settings"
"19.2.4 Setting Application process to be used"
"19.3 Customizing the Dashboard"
"19.4 Setting the Host Names for Virtual Servers"
"19.5 Setting the Method for Setting Resource Names"
"19.6 Settings for the Overcommit Function"
"19.7 Editing the Environment Setup File for the L-Platform API"
"19.9 System Disk Settings for Physical L-Servers"
"19.12 Edit the License Agreement"
"19.13 Editing the User Agreement when Registering a User"
After a Disaster Recovery environment is configured, if the settings of the primary site are changed according to "Chapter 8 Modifying Settings" in the "Operation Guide CE", reflect the changes of the following settings on the backup site.
"8.2 Settings for Sending Email"
"8.4 Editing Information in the Home Window"
"8.5.1 Settings for Permissions to Change L-Platform Templates"
"8.5.2 Subnet Settings at Segment Editing"
"8.5.3 Settings for the Simplified Reconfiguration Function"
"8.5.4 Distribution Ratio Settings"
"8.5.5 Application Process Settings"
"8.5.7 Edit the License Agreement"
"8.5.9 Default Password Setting for Sent Emails"
"8.5.10 Settings for the Maximum Number of Connections for the L-Platform Template"
"8.6.1 Settings for Tenant Management and Account Management"
"8.6.2 Editing the User Agreement when Registering a User"
"8.7 Accounting Settings"
"8.8 System Condition Server List Settings"
"8.9 Settings for Event Log Output for CMDB Agent"
Batch import
Perform batch switchover operations, using the switchover information for Disaster Recovery.
Operations to Perform Batch Switchover of All Resources
Execute the following command when settings for disaster recovery are complete.
[Windows Manager]
>Installation_folder\SVROR\Manager\bin\rcxrecovery <RETURN> |
Operation Involving Resource Switchover by Tenant
Restore global resources
Execute the following command when settings for disaster recovery are complete.
[Windows Manager]
>Installation_folder\SVROR\Manager\bin\rcxrecovery -global <RETURN> |
Restore each tenant resource
Execute the following command for each tenant.
Execute the following command when settings for disaster recovery are complete.
[Windows Manager]
>Installation_folder\SVROR\Manager\bin\rcxrecovery -tenant tenant1 <RETURN> |
For details on the rcxrecovery command, refer to "A.2 rcxrecovery".
The rcxrecovery command displays messages which represent the status of the switchover process. In addition, the following operations require manual operations or decisions by an administrator. Respond to inquiries from the system to continue execution of the command.
When the -nocleanup option is not specified and the backup site is to be cleaned up, an inquiry will be displayed to ask if you really want it to be cleaned up. Press "y" to approve the cleanup and continue the rcxrecovery command. If cleanup of the backup site cannot be performed, press "n" to quit the command.
When VM management software is not configured on the physical L-Server, manual restoration of VM management software is necessary. The rcxrecovery command will show a prompt while waiting for the restoration of VM management software. In that case, continue the command after restoration of VM management software is completed.
If VMware ESXi is installed on the server that is the target of switchover, the VM host must be registered in SVOM. After registration is complete, continue the command execution. The command will be postponed until registration is complete.
Information
The rcxrecovery command performs the following:
Mapping of storage
Storage resource mapping is performed, based on the files defining the relations between the storage on the primary and backup sites.
Division of import resources
Imported resources are divided using the following units:
- Global resources and resources of individual tenants
- Physical L-Server resources and virtual L-Server resources
Delete the backup site environment
Delete the following environment used for backup site operations.
- L-Platform templates
- L-Platform configuration information
- Resource information
- Accounting information
- Metering logs
Import the information
Import the following information:
- L-Platform templates
- L-Platform configuration information
- Resource information
- Accounting information
- Metering logs
Note
Configured physical servers and linked L-Servers for which only configuration definition has been created are not covered by disaster recovery failover.
Execute the rcxadm config command as follows to back up the information which is used to relate the configuration definition to L-Server again later.
[Windows Manager]
>Installation_folder\SVROR\Manager\bin\rcxadm config filter -convert -indir indir -outdir outdir <RETURN> |
The following information will be output to outdir, using the file name convert.txt.
After completing the restoration procedure, perform creation of a link with an L-Server and setting the scope again as shown below:
[command] |
Storage using the ETERNUS dynamic LUN mirroring is switched to the LUN that was prepared on the backup site when the switchover operation is performed after a disaster. Operation may be different if the storage is not released even after deleting an L-Platform.
Perform operation so that the import of the resource configuration information and user definition file (XML file) using the rcxrecovery command is not executed at the same time as modification of the resource configuration information or user definition file (XML file) due to L-Server creation, or export operations using the rcxmgrexport command.
The RC console may not display the correct power status of L-Servers immediately after a disaster recovery failover. Wait for about 3 minutes after the failover to see the correct power status of L-Servers.
As the performance information (the information displayed in the dashboard and in operational status displays) depends on the configuration of the primary site, this is not carried over to the backup site when switched over. The information on the backup site is displayed.
Importing may fail when switchover is performed to the backup site, due to a VM guest switchover failure. In this case, perform the import again using the following procedure, but excluding the managed servers for which import failed. Disaster recovery failover fails when configuration import fails. In this case, execute the failover command again after taking the corrective action for the cause of the import failure. When you execute the command again, the -nocleanup option needs to be specified if the previous command was executed without the -nocleanup option and configuration cleanup was performed and the "cleanup of resources" process ended in "completed".
The following shows an example of where the VM guest could not be restored during L-Server import.
1) An error occurs during import.
>Installation_folder\SVROR\Manager\bin\rcxrecovery <RETURN> FJSVrcx:ERROR:62569:/tenant1/l-platform1/l-server1:lserver:import was interrupted. Message=:67154: VM Guest not found |
2) The status of the relevant resources is displayed to help determine the cause of the error.
>Installation_folder\SVROR\Manager\bin\rcxadm config show -type lserver -name /tenant1/l-platform1/l-server1 -dir dir1 -format xml<RETURN> |
For -dir, specify the folder that was most recently exported in the folder below.
rcxrecovery -dir option, or the import folder specified in fa_dr.rcxprop\ManagerExport\RORexport_date
3) Delete the resource where the problem occurred from the export file.
>Installation_folder\SVROR\Manager\bin\rcxadm config filter -exclude lserver -name /tenant1/l-platform1/l-server1 -indir dir1 -outdir dir2 <RETURN> |
4) Use the export file output in 3) to overwrite the file in the folder specified for -dir in 2).
5) Import again.
>Installation_folder\SVROR\Manager\bin\rcxrecovery -nocleanup<RETURN> |
Restore directory service information
Restore the directory service information. For the restoration procedure, refer to the manuals of the directory service software.
Check the imported L-Platforms
When an L-Platform is exported during its deployment or a reconfiguration, and then the L-Platform configuration information is imported using the rcxrecovery command, L-Platform information without resources may be recovered.
In this case, information of the L-Platform whose resources may not exist is output to the log.
Confirm the details output in logs, and take corrective action.
For information on the content output to the log and the corrective actions, refer to "Appendix B Output Logs when Switchover Occurs because of Disaster Recovery".
Recover usage charges
Send usage charge files
Usage charge files for each tenant of the exported month will be recovered to the following directory on the primary site. Refer to "15.4.6 Sending Usage Charges" in the "Operation Guide CE" for the format of the usage charge files.
Usage Charge Files
Stored folder
Installation_folder\RCXCTMG\Charging\importdata
File Name
- YYYYMM_Tenant_name[_Deleted_date].zip
- YYYYMM is the cutoff year and month of the tenant.
- Deleted_date will be appended if the tenant is deleted.
These usage charge files will be needed for the accounts manager to determine the usage charges for each tenant on the month of the recovery.
Send these files to the accounts manager of each tenant based on the tenant information file. The tenant information file is a file describing the tenant and it holds information such as tenant name, cutoff date, accounting E-mail address and so on. The tenant information file is stored in the same directory as the usage charge files.
Tenant Information File
Stored folder
Installation_folder\RCXCTMG\Charging\importdata
File Name
tenant_info.csv
Format
CSV file with "Tenant name", "Tenant display name", "Tenant deleted date", "Tenant last cutoff date", "Tenant next cutoff date", and "Accounting E-mail address" columns.
#TenantName,TenantDisplayName,TenantDeletedDate,TenantLastCutoffDate,TenantNextCutoffDate,AccountingMailAddress tenant01,tenant001,,2012-05-31,2012-06-30,tenant1@example.com tenant02,tenant002,,2012-05-31,2012-06-30,tenant2@example.com tenant03,tenant003,2012-05-27 16:48,2012-05-31,2012-06-30,tenant3@example.com tenant04,tenant004,,2012-05-31,2012-06-30,tenant4@example.com |
Send usage charge files to the accounts manager as follows.
Operations to Perform Batch Switchover of All Resources
Send the recovered usage charge files of each tenant to the accounts manager. For each usage charge file, which tenant it corresponds to can be determined from the file name. Obtain the accounting E-mail addresses from the tenant information file and send the usage charge files.
Operation Involving Resource Switchover by Tenant
Obtain the accounting E-mail addresses of the switchover tenant from the tenant information file and send the target usage charge files.
When sending the usage charge files, also notify the recipient of the following information.
The usage files of the backup site will be sent separately on the day after the cutoff date.
Usage charges for the month of the recovery have to be calculated with the usage charge files of the affected site and the backup site.
In cases such as when the accounting calculation is stopped due to the disaster on the day of the cutoff of the tenant or the tenant has already been deleted, there is no accumulated data on the backup site. Therefore, the usage charge files will not be sent from the backup site. Notify the accounts manager that those tenants can be charged only from the usage charge files from the primary site. The cutoff date of the tenant can be obtained from tenant information file.
Notify the tenant administrator
In the Usage Charge window of the backup site, the usage charges are those of the backup up site only. Therefore, the infrastructure administrator has to inform tenant administrators of this fact, and if necessary, disclose the usage charge information of the primary site.
Note
Note that the calculation of the usage charge for the backup site will start the day following the recovery.
Change the FQDN setting
When admin servers have different IP addresses between the primary and backup sites, and the FQDN needs to be taken over by the backup site, change the IP address for the FQDN registered in DNS server to the IP address of the admin server on the backup site after disaster recovery failover is performed.
When the FQDN is taken over from the primary site to the backup site, clear the caches or temporary internet files of Web browsers which are used to connect to the ROR console on clients.
When test certificates are used, import the certificate of the backup site to the browser if the certificate was not imported during installation.
For details, refer to "Chapter 6 Importing a Certificate to a Browser" in the "Setup Guide CE".
The access rights that were modified on the primary site are not carried over to the backup site. Update them manually when the switchover is complete.
