Top
PRIMECLUSTER Global Disk Services  Configuration and AdministrationGuide 4.7
Chapter 7 Operation and Maintenance > 7.3 Disk Swap > 7.3.3 Swapping Disks of Netmirror Group
PreviousNext

7.3.3 Swapping Disks of Netmirror Group

7.3.3.1 Hot Swap

To perform hot swap (swap without stopping the system) of a disk connected to the netmirror group, perform the following procedure.

  1. Make a disk exchangeable.

    • For shared classes

      Execute "Swap Physical Disk" (or the sdxswap -O command) on the node where a disk is to be swapped.

      If multiple partitions on the disk to be swapped are the targets of the mirroring among servers, execute "Swap Physical Disk" (or the sdxswap -O command) for iSCSI devices for all the partitions.

    • For local classes (when object configurations were restored with the sdxconfig Restore command)

      Delete the entire object configuration (volumes, groups, and disks) of a local class to delete the local class.

      As for the order of deleting disks, delete a disk to be swapped with the sdxdisk -R command first.

      If multiple partitions on the disk to be swapped are the targets of the mirroring among servers, delete the object configurations for iSCSI devices for all the partitions.

      Note

      If a netmirror group exists in a local class, "Swap Physical Disk" (or the sdxswap -O command) cannot be executed.

  2. Stop iSCSI devices for disks to be swapped on both nodes.

    If multiple partitions on disks to be swapped are the target of the mirroring among servers, execute it for iSCSI devices for all the partitions.

    # echo offline > /sys/block/iSCSI_device_name/device/state

    Execution example

    # echo offline > /sys/block/sdd/device/state

  3. Back up the target information on a node where a disk is swapped.

    # cp /etc/target/saveconfig.json backup_file

    Execution example

    # cp /etc/target/saveconfig.json /var/tmp/targetfile.json

  4. Confirm storage and LUN for a device to be swapped on a node where a disk is swapped.

    If multiple partitions on disks to be swapped are the target of the mirroring among servers, confirm all the partitions.

    # targetcli ls

    Output example

    # targetcli ls
o- / .............................................................................. [...]
  o- backstores ................................................................... [...]
  | o- block ....................................................... [Storage Objects: 2]
  | | o- storage1 . [/dev/disk/by-id/scsi-3500000e114784363 (8.0GiB) write-thru activated] (*1)
  | | o- storage2.[/dev/disk/by-id/scsi-3500000e1109cc830 (16.0GiB) write-thru activated]
  | o- fileio ...................................................... [Storage Objects: 0]
  | o- pscsi ....................................................... [Storage Objects: 0]
  | o- ramdisk ..................................................... [Storage Objects: 0]
  o- iscsi ................................................................. [Targets: 1]
  | o- iqn.2003-01.org.linux-iscsi.node1.x8664:sn.6e665c7c1be0 ................ [TPGs: 1]
  |   o- tpg1 .................................................... [no-gen-acls, no-auth]
  |     o- acls ............................................................... [ACLs: 2]
  |     | o- iqn.1994-05.com.redhat:dee92ff9979d ....................... [Mapped LUNs: 2]
  |     | | o- mapped_lun0 .................................... [lun0 block/storage1 (rw)] (*2)
  |     | | o- mapped_lun1 ................................... [lun1 block/storage2 (rw)]
  |     | o- iqn.1994-05.com.redhat:fa7eb9cf483c ....................... [Mapped LUNs: 2]
  |     |   o- mapped_lun0 .................................... [lun0 block/storage1 (rw)] (*2)
  |     |   o- mapped_lun1 ................................... [lun1 block/storage2 (rw)]
  |     o- luns ............................................................... [LUNs: 2]
  |     | o- lun0 .............. [block/storage1 (/dev/disk/by-id/scsi-3500000e114784363)] (*3)
  |     | o- lun1 ............. [block/storage2 (/dev/disk/by-id/scsi-3500000e1109cc830)]
  |     o- portals ......................................................... [Portals: 1]
  |       o- 192.168.125.123:3260 .................................................. [OK]
  o- loopback .............................................................. [Targets: 0]

    (*1) Confirm that the storage name for /dev/disk/by-id/scsi-3500000e114784363 is "storage1".

    (*2) It is LUN that is mapped to the initiator allowed to connected to. The information related to lun0 was output to the line.

    (*3) It shows that the storage name for lun0 is "storage1" and the by-id file of the device is /dev/disk/by-id/scsi-3500000e114784363.

    In the example, if you swap the device indicated by /dev/disk/by-id/scsi-3500000e114784363, the storage name is "storage1" and LUN is "lun0".

  5. Delete all the LUNs to be swapped on a node where a disk is to be swapped.

    If multiple partitions on disks to be swapped are the target of the mirroring among servers, execute this step for storages for all the partitions.

    # targetcli /backstores/block/ delete storage_name

    Point

    Specify the storage name confirmed in step 4 to storage_name.

    Execution example

    # targetcli /backstores/block/ delete storage1
Deleted storage object storage1.

  6. Delete storage and LUN for a device to be swapped on a node where a disk is to be swapped.

    If multiple partitions on disks to be swapped are the target of the mirroring among servers, confirm storage and LUN for all the partitions.

    # targetcli ls

    Point

    Confirm that the information of the deleted storage and LUN is not output.

    Output example

    # targetcli ls
o- / ................................................................................ [...]
  o- backstores ..................................................................... [...]
  | o- block ......................................................... [Storage Objects: 1]
  | | o- storage2 . [/dev/disk/by-id/scsi-3500000e1109cc830 (16.0GiB) write-thru activated]
  | o- fileio ........................................................ [Storage Objects: 0]
  | o- pscsi ......................................................... [Storage Objects: 0]
  | o- ramdisk ....................................................... [Storage Objects: 0]
  o- iscsi ................................................................... [Targets: 1]
  | o- iqn.2003-01.org.linux-iscsi.node1.x8664:sn.6e665c7c1be0 .................. [TPGs: 1]
  |   o- tpg1 ...................................................... [no-gen-acls, no-auth]
  |     o- acls ................................................................. [ACLs: 2]
  |     | o- iqn.1994-05.com.redhat:dee92ff9979d ......................... [Mapped LUNs: 1]
  |     | | o- mapped_lun1 ..................................... [lun1 block/storage2 (rw)]
  |     | o- iqn.1994-05.com.redhat:fa7eb9cf483c ......................... [Mapped LUNs: 1]
  |     |   o- mapped_lun1 ..................................... [lun1 block/storage2 (rw)]
  |     o- luns ................................................................. [LUNs: 1]
  |     | o- lun1 ............... [block/storage2 (/dev/disk/by-id/scsi-3500000e1109cc830)]
  |     o- portals ........................................................... [Portals: 1]
  |       o- 192.168.125.123:3260 .................................................... [OK]
  o- loopback ................................................................ [Targets: 0]

  7. Swap the disk.

    For the procedures for swapping disks, refer to the manuals related to the servers and the disks.

  8. If the mirroring among servers is performed not per the physical disk but per partition, create partitions on the swapped disk on the node where the disk was swapped.

    The partition size and the order to create the partition must be consistent with the settings before and after the disk is swapped.

    In an Azure environment, a NIFCLOUD environment, or an environment where the by-id file does not exist, set the gpt disk label in the swapped disk, and create a partition.

    The partition size must be consistent with the size before the disk is swapped.

  9. Edit the target information that was backed up in step 3 and modify the path of the by-id file for the original disk to that for the new disk.

    If multiple partitions on the disk to be swapped are the targets of the mirroring among servers, take this step for the path of the by-id file for all the partitions.

    In an Azure environment, a NIFCLOUD environment, or an environment where the by-id file does not exist, modify the path of the by-partuuid file for the original disk to by-partuuid for the partition of the new disk.

    # vim backup_file

{
  "fabric_modules": [],
  "storage_objects": [
    {
...
      "dev": "original_disk",    (*1)
      "name": "storage_name",
      "plugin": "block",
      "readonly": false,
      "write_back": false,
      "wwn": "fae93a41-0d59-4843-9137-9c39854e5388"
    }
  ],

    (*1) Modify to the path of the by-id file or the path of the by-partuuid file for the new disk.

    Point

    Specify the backup file that was backed up in step 3 to backup_file.

    [Example before modification]

    # vim /var/tmp/targetfile.json

{
  "fabric_modules": [],
  "storage_objects": [
    {
...
      "dev": "/dev/disk/by-id/scsi-3500000e114784363",
      "name": "storage1",
      "plugin": "block",
      "readonly": false,
      "write_back": false,
      "wwn": "279b8505-418f-4449-8987-f3acaad34618"
    }
  ],

    [Example after modification]

    # vim /var/tmp/targetfile.json

{
  "fabric_modules": [],
  "storage_objects": [
    {
...
      "dev": "/dev/disk/by-id/scsi-3500000e111c56611",
      "name": "storage1",
      "plugin": "block",
      "readonly": false,
      "write_back": false,
      "wwn": "279b8505-418f-4449-8987-f3acaad34618"
    }
  ],

  10. Restore the target information that was edited in step 9 on a node where a disk is to be swapped.

    # targetctl restore backup_file

    Point

    Specify the backup file that was created in step 9 to backup_file.

    Execution example

    # targetctl restore /var/tmp/targetfile.json

  11. Confirm that the storage and LUN was added to a node where a disk is to be swapped.

    If multiple partitions on the disk to be swapped are the targets of the mirroring among servers, confirm the storage and LUN for all the partitions.

    # targetcli ls

    Point

    (*1) Confirm that the device correspond to the storage that was deleted in step 5 is changed to the swapped device.

    (*2) Confirm that LUN which was deleted in step 5 is mapped to the initiator allowed to connected to.

    (*3) Confirm the following:

    - LUN and the storage that were deleted in step 5 are added.

    - The path of the device is specified to the device that was swapped.

    Output example

    # targetcli ls
o- / .............................................................................. [...]
  o- backstores ................................................................... [...]
  | o- block ....................................................... [Storage Objects: 2]
  | | o- storage1 . [/dev/disk/by-id/scsi-3500000e111c56611 (8.0GiB) write-thru activated] (*1)
  | | o- storage2.[/dev/disk/by-id/scsi-3500000e1109cc830 (16.0GiB) write-thru activated]
  | o- fileio ...................................................... [Storage Objects: 0]
  | o- pscsi ....................................................... [Storage Objects: 0]
  | o- ramdisk ..................................................... [Storage Objects: 0]
  o- iscsi ................................................................. [Targets: 1]
  | o- iqn.2003-01.org.linux-iscsi.node1.x8664:sn.6e665c7c1be0 ................ [TPGs: 1]
  |   o- tpg1 .................................................... [no-gen-acls, no-auth]
  |     o- acls ............................................................... [ACLs: 2]
  |     | o- iqn.1994-05.com.redhat:dee92ff9979d ....................... [Mapped LUNs: 2]
  |     | | o- mapped_lun0 .................................... [lun0 block/storage1 (rw)] (*2)
  |     | | o- mapped_lun1 ................................... [lun1 block/storage2 (rw)]
  |     | o- iqn.1994-05.com.redhat:fa7eb9cf483c ....................... [Mapped LUNs: 2]
  |     |   o- mapped_lun0 .................................... [lun0 block/storage1 (rw)] (*2)
  |     |   o- mapped_lun1 ................................... [lun1 block/storage2 (rw)]
  |     o- luns ............................................................... [LUNs: 2]
  |     | o- lun0 .............. [block/storage1 (/dev/disk/by-id/scsi-3500000e111c56611)] (*3)
  |     | o- lun1 ............. [block/storage2 (/dev/disk/by-id/scsi-3500000e1109cc830)]
  |     o- portals ......................................................... [Portals: 1]
  |       o- 192.168.125.123:3260 .................................................. [OK]
  o- loopback .............................................................. [Targets: 0]

  12. Store the target information that was restored in step 10 on a node where a disk is to be swapped.

    # targetctl save

  13. Rescan an iSCSI session on both nodes.

    # iscsiadm -m session --rescan

  14. Enable iSCSI devices on both nodes.

    If the mirroring among servers is performed per partition, enable iSCSI devices for all the partitions.

    # echo running > /sys/block/iSCSI_device_name/device/state

    Execution example

    # echo running > /sys/block/sdd/device/state

    At the time, the following OS message may be output to the system log. An action is not required.

    kernel: sd X:X:X:X: Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatically remap LUN assignments.

  15. Restore the GDS configuration.

    • For shared classes

      On the node where the disk was swapped, execute "Restore Physical Disk" (or the sdxswap -I command). Execute "Restore Physical Disk" (or the sdxswap -I command) for all the iSCSI devices where "Swap Physical Disk" (or the sdxswap -O command) was executed in step 1.

    • For local classes

      Re-create the object configuration (volumes, groups, disks, and classes) of the deleted local class.

      • When the configuration information file of the class exists

        When the class configuration information obtained with the sdxconfig Backup command exits, re-create the object configuration with the sdxconfig Restore command.

      • When the configuration information file of the class does not exist

        Specify a shared class to re-create the object configuration.

7.3.3.2 Cold Swap

To perform cold swap (swap in a stopped system) of a disk connected to the netmirror group, perform the following procedure.

  1. Make a disk exchangeable.

    • For shared classes

      Execute "Swap Physical Disk" (or the sdxswap -O command) on the node where a disk is to be swapped.

      If multiple partitions on the disk to be swapped are the targets of the mirroring among servers, execute "Swap Physical Disk" (or the sdxswap -O command) for iSCSI devices for all the partitions.

    • For local classes (when object configurations were restored with the sdxconfig Restore command)

      Delete the entire object configuration (volumes, groups, and disks) of a local class to delete the local class.

      As for the order of deleting disks, delete a disk to be swapped with the sdxdisk -R command first.

      If multiple partitions on the disk to be swapped are the targets of the mirroring among servers, delete the object configurations for iSCSI devices for all the partitions.

      Note

      If a netmirror group exists in a local class, "Swap Physical Disk" (or the sdxswap -O command) cannot be executed.

  2. If you swap the disk of the operating node while the service continues, switch the cluster application to make a node, which is not connected to the disk to be swapped, as an operating node.

  3. Modify the settings for starting the system on the node where the disk is to be swapped so that the system is restarted in single user mode.

    # systemctl set-default rescue.target

    Because the console cannot be used in a cloud environment, do not select single user mode. In this case, perform step 3-1 to 3-3 to restrict the automatic startup of the PRIMECLUSTER service.

    In an environment other than a cloud environment, follow the procedure from step 4.

    1. Stop RMS.

      If RMS is running, execute the following command on any one node to stop RMS.

      # hvshut -a

      Note

      If RMS is stopped on all nodes during the synchronization copying of the GDS volume, the synchronization copying of the entire volume area is performed after the corrections are applied and all nodes are restarted.

      If you do not want to perform the synchronization copying of the entire area of volume, stop RMS after the synchronization copying is completed.

      To check the slice status of the GDS volume, execute the following command.

      Execute the following command on any one node to check the value of the STATUS field of the command output.

      The status of the copy destination slice is COPY during the synchronization copying, and after copying is complete, the status becomes ACTIVE or STOP.

      # sdxinfo -S

    2. Restrict the automatic startup of the PRIMECLUSTER service.

      Restrict the automatic startup of the PRIMECLUSTER service by executing the following command on the node where the disk is to be swapped.

      # /opt/FJSVpclinst/bin/pclservice off

    3. Restart the system.

      Restart the system on the node where the disk is to be swapped.

      # /sbin/shutdown -r now

  4. Stop the node where the disk is to be swapped.

  5. Swap the disk.

    For the procedures for swapping the disk, refer to the manuals related to the servers and the disks.

  6. Start the node where the disk was swapped.

  7. If the mirroring among servers is performed not per the physical disk but per partition, create partitions on the swapped disk on the node where the disk was swapped.

    The partition size and the order to create the partition must be consistent with the settings before and after the disk is swapped.

    In an Azure environment, a NIFCLOUD environment, or an environment where the by-id file does not exist, set the gpt disk label in the swapped disk, and create a partition.

    The partition size must be consistent with the size before the disk is swapped.

  8. Back up the setting information of iSCSI target on the node where the disk was swapped.

    # cp /etc/target/saveconfig.json backup_file

    Execution example

    # cp /etc/target/saveconfig.json /var/tmp/targetfile.json

  9. Modify the path of the by-id file for the original disk to that for the new disk in the setting information of the iSCSI target.

    If multiple partitions on the disk to be swapped are the targets of the mirroring among servers, take this step for the path of the by-id file for all the partitions.

    In an Azure environment, a NIFCLOUD environment, or an environment where the by-id file does not exist, modify the path of the by-partuuid file for the original disk to by-partuuid for the partition of the new disk.

    # vim /etc/target/saveconfig.json

{
  "fabric_modules": [],
  "storage_objects": [
    {
...
      "dev": "original_disk",  (*1)
      "name": "storage_name",
      "plugin": "block",
      "readonly": false,
      "write_back": false,
      "wwn": "fae93a41-0d59-4843-9137-9c39854e5388"
    }
  ],

    (*1) Modify to the path of the by-id file or the path of the by-partuuid file for the new disk.

    [Example before modification]

    # vim /etc/target/saveconfig.json

{
  "fabric_modules": [],
  "storage_objects": [
    {
...
      "dev": "/dev/disk/by-id/scsi-3500000e111e68e00",
      "name": "store1",
      "plugin": "block",
      "readonly": false,
      "write_back": false,
      "wwn": "4a98bfb0-7d7e-4bc8-962c-0b3cf192b214"
    }
  ],

    [Example after modification]

    # vim /etc/target/saveconfig.json

{
  "fabric_modules": [],
  "storage_objects": [
    {
...
      "dev": "/dev/disk/by-id/scsi-3500000e111c56610", * The line to be modified
      "name": "store1",
      "plugin": "block",
      "readonly": false,
      "write_back": false,
      "wwn": "4a98bfb0-7d7e-4bc8-962c-0b3cf192b214"
    }
  ],

  10. Undo the settings for starting the system modified in step 3 so that the system is restarted in multi-user mode.

    Example) When setting back to multi-user.target

    # systemctl set-default multi-user.target

  11. Restart the node where the disk was swapped.

    In a cloud environment, take the following procedure to cancel restricting the automatic startup of the PRIMECLUSTER service.

    1. Configure the automatic startup of the PRIMECLUSTER service.

      Execute the following command on the node where the disk was swapped and change the PRIMECLUSTER service settings back to the state they were in before they were restricted in 2 of step 3.

      # /opt/FJSVpclinst/bin/pclservice on

    2. Restart the system.

      Restart the system on the node where the disk was swapped.

      # /sbin/shutdown -r now

  12. Check that iSCSI target has been set normally on the node where the disk was swapped.

    # targetcli /backstores/block/ ls ; targetcli /iscsi/ ls

    Point

    (*1) Confirm that the modified path is effective.

    Output example

    o- block ........................................................... [Storage Objects: 1]
  o- store1 ..... [/dev/disk/by-id/scsi-3500000e111c56610 (16.0GiB) write-thru activated] (*1)
o- iscsi ................................................................... [Targets: 1]
  o- iqn.2003-01.org.linux-iscsi.node1.x8664:sn.6e665c7c1be0 .................. [TPGs: 1]
    o- tpg1 ...................................................... [no-gen-acls, no-auth]
      o- acls ................................................................. [ACLs: 2]
      | o- iqn.1994-05.com.redhat:dee92ff9979d ......................... [Mapped LUNs: 1]
      | | o- mapped_lun0 ....................................... [lun0 block/store1 (rw)]
      | o- iqn.1994-05.com.redhat:fa7eb9cf483c ......................... [Mapped LUNs: 1]
      |   o- mapped_lun0 ....................................... [lun0 block/store1 (rw)]
      o- luns ................................................................. [LUNs: 1]
      | o- lun0 ................. [block/store1 (/dev/disk/by-id/scsi-3500000e111c56610)] (*1)
      o- portals ........................................................... [Portals: 2]
        o- 192.168.56.20:3260 ...................................................... [OK]

  13. On the node where the disk was swapped, check the state of the iSCSI session.

    Check that both the iSCSI connection state and the session state indicate "LOGGED IN".

    Execution example

    # iscsiadm -m session -P 1
...
Target: iqn.2003-01.org.linux-iscsi.node1.x8664:sn.6e665c7c1be0
        Current Portal: 192.168.56.20:3260,1
        Persistent Portal: 192.168.56.20:3260,1
                **********
                Interface:
                **********
                ...
                iSCSI Connection State: LOGGED IN
                iSCSI Session State: LOGGED_IN
                ...

  14. Rescan an iSCSI session on both nodes.

    # iscsiadm -m session --rescan

  15. Restore the GDS configuration.

    • For shared classes

      On the node where the disk was swapped, execute "Restore Physical Disk" (or the sdxswap -I command). Execute "Restore Physical Disk" (or the sdxswap -I command) for all the iSCSI devices where "Swap Physical Disk" (or the sdxswap -O command) was executed in step 1.

    • For local classes

      Re-create the object configuration (volumes, groups, disks, and classes) of the deleted local class.

      • When the configuration information file of the class exists

        When the class configuration information obtained with the sdxconfig Backup command exits, re-create the object configuration with the sdxconfig Restore command.

      • When the configuration information file of the class does not exist

        Specify a shared class to re-create the object configuration.

  16. Delete the backup file created in step 8.

    # rm backup_file

    Execution example

    # rm /var/tmp/targetfile.json
PreviousNext
Copyright Fujitsu Limited 2023