This chapter explains the procedure for switchover operations in a Disaster Recovery environment when the system has been damaged.
When you change from backup site to operation site, you can switchover same operations.
Use the following procedure for the switchover in a Disaster Recovery environment:
Edit resource configuration information and user definition information (XML files)
Commands must be used to edit the Resource Configuration Information and User Definition Information (XML files), such as dividing into units for import or changing resource information to match the backup site environment.
Use the following procedure to edit the configuration information.
Separating an L-Server linked with a configured physical server
L-Servers linked to configured physical servers cannot be restored using Disaster Recovery.
After performing the following operations, execute the command to create a link with an L-Server. Once the restoration is complete, create a link with the L-Server again.
>rcxadm config filter -convert -indir indir -outdir outdir <RETURN> |
The following information will be output to outdir, using the file name convert.txt.
After completing the restoration procedure, perform creation of a link with an L-Server and setting the scope again as shown below:
[command] |
Mapping of storage
Create files to perform mapping of the storage information of the backup site from the storage replication information created in step 7. of "15.1 Installation".
For mapping file creation, execute the rcxstorage command with the -recovery and -mode options. Specify "mapping" for the -mode option.
For details on the rcxstorage command, refer to "1.7.13 rcxstorage" of the "Reference Guide (Resource Management) CE".
Change the storage information of the Resource Configuration Information and User Definition Information (XML files) of Resource Orchestrator imported using the created mapping files. Execute the rcxadm config map command to change the storage information.
For details on the rcxadm config map command, refer to "1.7.11 rcxadm config" in the "Reference Guide (Resource Management) CE".
Note
Storage using the ETERNUS dynamic LUN mirroring is switched to the LUN that was prepared on the backup site when the switch operation is performed after a disaster. Operation may be different if the storage is not released even after deleting the L-Platform.
Division of Tenant Units
When operating after creating tenants, it is necessary to divide the Resource Configuration Information and User Definition Information (XML files) to import for Resource Orchestrator into information common among tenants and information unique to each tenant. Execute the rcxadm config filter command to divide Resource Configuration Information and User Definition Information (XML files) relating to tenants for Resource Orchestrator.
For details on the rcxadm config filter command, refer to "1.7.11 rcxadm config" in the "Reference Guide (Resource Management) CE".
When switchover operation each tenant with damaged
Resources shared between tenants
Execute the command with the -global option specified.
Resources of individual tenants
Execute the command for each tenant, with the -tenant option specified. Resource Configuration Information will be divided for each tenant.
When switchover operation managed server with damaged
Execution of the rcxadm config filter command is unnecessary because dividing is unnecessary.
Division of virtual L-Servers and physical L-Servers
When operating using a virtual L-Server, the virtual L-Server must be restored after restoring the physical L-Server that VM management software is mounted on. Therefore, it is necessary to divide the virtual L-Server-related resources and other resources. Regarding virtual L-Server division, perform the operation in the following environments:
Environments not using tenants
Environments shared between tenants divided in step c.
Environments for individual tenants
Use the rcxadm config filter command for this operation.
For details on the rcxadm config filter command, refer to "1.7.11 rcxadm config" in the "Reference Guide (Resource Management) CE".
Stop operation of the backup site
When using other operations on the backup site, stop the L-Platform operations and delete the L-Platform information and the following resources used by each L-Platform.
L-Platform template
L-Platform configuration information
Resource information
Accounting information
Metering log
The procedures for deleting the different pieces of information are as following.
Delete the configuration information of resource management.
The deletion of the configuration information of the resource management stops the business of L-Platform, and deletes the resource of the following resource managements used with the L-Platform deletion and each L-Platform for management.
L-Server
L-Server Templates
Tenant Folders
Resource Folders
Resource Pool
Image resource
Users
User Groups
Network resource
Address Set resource
Note
It is not possible to delete all resource pools for backup sites. Ensure that at least one resource remains in the resource pool specifically for the backup site, and ensure that its name does not overlapping with the names of the primary site and other resource pools.
This resource pool is not required after the import. Delete the resource pool when switchover operations after damage are completed.
Ensure there is at least one user remaining, such as an infrastructure administrator or a dual-role administrator, to perform import operations. When using the same user name as a remaining user on the primary site, settings for the backup site are valid even after import. For instance, the password will not change to the password used in the primary site. The password originally used in the backup site can be used.
Delete metering log information.
Execute the ctchg_deletemeterlog as follows to delete the metering log information.
Modify the following configuration file for metering log operations:
File | Windows | Installation_folder\RCXCTMG\Charging\conf\metering_log.properties |
Linux | /opt/FJSVctchg/conf/metering_log.properties | |
Changes | retention_period=0000-00-00 | |
Execute the ctchg_deletemeterlog command.
For details on the ctchg_deletemeterlog command, refer to "3.2 ctchg_deletemeterlog (Delete Metering Logs)" in the "Reference Guide CE".
Restore the settings in the operation settings file for the metering log that were changed in 1.
The default is retention_period=0000-03-00.
Disconnect the storage
Stop the storage replication function, and disconnect the backup site from the primary site.
Copy firewall configurations
When using the firewall auto-configuration function for network devices, copy the newest configuration from the primary site to the firewall of the backup site.
Copy execution information files of sample scripts
Copy information backed up on the operation site under the control of the following folders of the backup site.
Installation_folder\SVROR\Manager\etc\scripts
Refer to "Backup of execution information in the sample script" of "15.2 Normal Operation" for information backed up on the primary site.
Modify the server NIC definition
When the server NIC definition has been changed and reflected on the manager on the primary site, change the server NIC definition on the backup site, and reflect it on the manager.
Execute the rcxadm nicdefctl commit command, and reflect the modified server NIC definition on the manager.
For details on the server NIC definition, refer to "2.11 Server NIC Definitions" of the "Reference Guide (Resource Management) CE".
For details on the rcxadm nicdefctl commit command, refer to "1.7.16 rcxadm nicdefctl" in the "Reference Guide (Resource Management) CE".
Import appropriate information and start L-Servers
Import the Resource Configuration Information and User Definition Information (XML files) for the backup site. Manually start the created L-Platform using the import operation, as it has been stopped.
Note
When this procedure are done, refer the note in "section 2 a" and delete the resource pools that were not deleted yet.
When importing resource configuration information and user definition information by executing rcxadm config import. As not all resources are registered after testing, only some resources may be registered. In this case, import again after dealing with the cause of failures.
When executing import, the following are automatically performed.
Resource folder creation (tenant folders)
Resource pool creation
Registration of resources in resource pools
L-Server template creation
System image information creation
Snapshot information creation
L-Server creation
Firewall creation
User group creation
User creation
Operate importing of the resource configuration information and user definition information (XML file) at the same time as operating it so as not to execute it the change of the resource configuration information and user definition information (XML file) such as making L-Server.
Importing may fail when switchover is performed to the backup site, due to a VM guest switchover failure. In this case, perform the import again using the following procedure, but excluding the managed servers for which import failed.
The following shows an example of where the VM guest could not be restored during L-Server import.
An error has occurred during import.
>rcxadm config import -dir dir1 <RETURN>
FJSVrcx:ERROR:62569:/tenant1/l-platform1/l-server1:lserver:import was interrupted. Message=:67154: VM Guest not found |
The status of the relevant resources is displayed to help determine the cause of the error.
>rcxadm config show -type lserver -name /tenant1/l-platform1/l-server1 -dir dir1 -format xml<RETURN> |
Delete the resource where the problem occurred from the export file.
>rcxadm config filter -exclude lserver -name /tenant1/l-platform1/l-server1 -indir dir1 -outdir dir2 <RETURN> |
Import again.
>rcxadm config import -dir dir2 <RETURN> |
Import the resource configuration information and user definition information (XML file).
Execute the rcxadm config import command to import Resource Configuration Information and User Definition Information (XML files).
For details on the rcxadm config import command, refer to "1.7.11 rcxadm config" in the "Reference Guide (Resource Management) CE".
Execute import of resources other than virtual L-Servers.
The imported L-Server is in the status where it has only been defined.
Start the imported L-Server.
Start a physical L-Server with VM management software mounted and a physical L-Server with a VM host mounted. There are no problems starting other L-Servers after performing step e.
Register VM management software in the resource tree if necessary.
It is not necessary to registration again, when registration has already been performed on another tenant, sharing VM management software with other tenants. When registering VM management software, register using the same name as that of the primary site.
Update VM management software information, and reflect the information.
Execute the rcxadm vmmgr command to update the VM management software information.
For details on the rcxadm vmmgr command, refer to "1.7.3 rcxadm vmmgr" of the "Reference Guide (Resource Management) CE".
Import resources related to virtual L-Servers.
The imported L-Server is in the status where it has only been defined.
When import is executed, a test of whether importing for the Resource Configuration Information and User Definition Information (XML files) is available or not is performed. If available, an L-Server is created, then the resources allocated, and the L-Server is ready for use.
Deletes the L-Platform template and the L-Platform configuration information.
Execute the cfmg_deleteconfig command to delete the L-Platform template and the L-Platform configuration information.
Refer to "5.11 cfmg_deleteconfig (Delete All Configuration Information)" in the "Reference Guide CE" for information on the cfmg_deleteconfig command.
Imports the L-Platform template and the L-Platform configuration information.
Execute the cfmg_importconfig command to import the L-Platform template and the L-Platform configuration information.
For details on the cfmg_importconfig command, refer to "5.10 cfmg_importconfig (Import Configuration Information)" in the "Reference Guide CE".
Imports the accounting information.
To import accounting information, execute the productmaintain import feature of the product maintenance command.
For details on the command for the maintenance of the product master, refer to "3.4 productmaintain (Product Master Maintenance)" in the "Reference Guide CE".
Restores the directory service information.
Restores the directory service information.
Copy global resources.
Import the resource configuration information and user definition information (XML file).
Execute the rcxadm config import command to import Resource Configuration Information and User Definition Information (XML files).
For details on the rcxadm config import command, refer to "1.7.11 rcxadm config" in the "Reference Guide (Resource Management) CE".
Execute import of resources other than virtual L-Servers.
The imported L-Server is in the status where it has only been defined.
Start the imported L-Server.
Start a physical L-Server with VM management software mounted and a physical L-Server with a VM host mounted. There are no problems starting other L-Servers after performing step e.
Register VM management software in the resource tree if necessary.
It is not necessary to registration again, when registration has already been performed on another tenant, sharing VM management software with other tenants. When registering VM management software, register using the same name as that of the primary site.
Update VM management software information, and reflect the information.
Execute the rcxadm vmmgr command to update the VM management software information.
For details on the rcxadm vmmgr command, refer to "1.7.3 rcxadm vmmgr" of the "Reference Guide (Resource Management) CE".
Deletes the L-Platform template and the L-Platform configuration information.
Execute the cfmg_deleteconfig command to delete the L-Platform template and the L-Platform configuration information.
Refer to "5.11 cfmg_deleteconfig (Delete All Configuration Information)" in the "Reference Guide CE" for information on the cfmg_deleteconfig command.
Import global resource.
Import the global L-Platform template and L-Platform configuration information.
The L-Platform template of global and importing of the L-Platform configuration information are executed specifying to the cfmg_importconfig command "-global" optional.
For details on the cfmg_importconfig command, refer to "5.10 cfmg_importconfig (Import Configuration Information)" in the "Reference Guide CE".
Imports the accounting information.
Execute the command for the maintenance of the product master as in the case of the procedure explained in "Procedure for Importing All Resources".
Restores the directory service information.
When tenants in the backup site requires to use the user accounts for the tenants that are not switched over from the primary site, delete these accounts first from directory service.
Import tenant resource.
The following procedures are executed to each importing tenant.
Execute the rcxadm config import command to import Resource Configuration Information and User Definition Information (XML files).
Execute import of resources other than virtual L-Servers.
The imported L-Server is in the status where it has only been defined.
Start the imported L-Server.
Start a physical L-Server with VM management software mounted and a physical L-Server with a VM host mounted. There are no problems starting other L-Servers after performing step e.
Register VM management software in the resource tree if necessary.
It is not necessary to registration again, when registration has already been performed on another tenant, sharing VM management software with other tenants. When registering VM management software, register using the same name as that of the primary site.
Update VM management software information, and reflect the information.
Execute the rcxadm vmmgr command to update the VM management software information.
For details on the rcxadm vmmgr command, refer to "1.7.3 rcxadm vmmgr" of the "Reference Guide (Resource Management) CE".
Import resources related to virtual L-Servers.
The imported L-Server is in the status where it has only been defined.
Imports the tenant's L-Platform template and the L-Platform configuration information.
To import the global L-Platform template and the L-Platform configuration information, execute the cfmg_importconfig command with the -tenant option.
For details on the cfmg_importconfig command, refer to "5.10 cfmg_importconfig (Import Configuration Information)" in the "Reference Guide CE".
Check the imported L-Platforms
When an L-Platform is exported during its deployment or a reconfiguration and then the L-Platform configuration information is imported using cfmg_importconfig, L-Platform information without resources may be recovered.
In this case, information of the L-Platform whose resources may not exist is output to the log.
Confirm the details output in logs, and take corrective action.
For information on the content output to the log and the actions, refer to "1.2 Log Output of Disaster Recovery Commands" in the "Reference Guide CE".
Update the operation status
Updates the configuration information used for collecting performance information.
Execute the cmdbrefresh command with the "-a" and "-q" options to update the configuration information used for collecting performance information.
For details on the cmdbrefresh command, refer to "5.12 cmdbrefresh (Refresh Configuration Information of System Condition)" in the "Reference Guide CE".
Note
As the performance information (the information displayed in the dashboard and in operational status displays) depends on the configuration of the main site, this is not carried over to the backup site when switched. The information on the backup site is displayed. The information on the backup site is displayed.
Merge metering logs
Accounting calculations can be made by merging the metering log exported from the main site and the metering log obtained (exported) from the backup site after the system is restored.
Note
Keep the metering log in its most recent condition as a backup so that the amount of information lost is limited when a fatal error occurs to the system, such as when a disaster occurs.
Start L-Servers
As the power status of the imported L-Server is off, perform the following procedure to turn the power of the necessary servers on.
Physical L-Servers created by infrastructure administrators can be started using the rcxadm folder start command or the rcxadm lserver start command.
When the L-Server belongs to an L-Platform, power on it using the StartLServer command or the StartLPlatform command.
