Dynamic Reconfiguration User's Guide I/O device edition - for Solaris (TM) Operating System -

3.1.2 Replacement of redundant configuration

When the PCI cards are needed to be replaced in redundant configuration with using software that controls the multipath, the PCI cards can be replaced without stopping the high-level products (services) as applications.

This section shows the replacement procedure of PCI cards for redundant configuration with using the following redundant software products.

• Multipath Disk Control (MPHD)
• Multipath Disk Control load balance option (MPLB)
• GR Multipath Driver (GRMPD)
• ETERNUS Multipath Driver (ETERNUS MPD)

In case of using the other redundant software products, refer to the manual for each product.

1. Stop the daemons

   If you use the Fibre Channel Card (PW028FC3*/PW028FC4*/PW028FC5*), execute the following.

   # /etc/rc0.d/K10ElxRMSrv stop <Return>
   # /etc/rc0.d/K10ElxDiscSrv stop <Return>

2. Locating the PCI cards and the system boards

   Using the following procedures, locate the system boards equipped with the faulty PCI cards and the all connected devices. The following procedure is described for MPHD/MPLB/GRMPD/ETERNUS MPD. If you use the redundancy software other than MPHD/MPLB/GRMPD/ETERNUS MPD, see the document for each product.
   If you use ETERNUS MPD, mplbt is displayed instead of hddv.

   1. Locate the path interfaces connected the faulty PCI cards with I/O devices from the WARNING messages information output to the console.
      The following example shows that fjpfca3 is the path inteface connected the faulty PCI card with I/O device.

      :
      WARNING: /pci@80,4000/fibre-channel@1 (fjpfca3):
      Hard Error : PCI DMA error.
      :
      WARNING: /pci@80,4000/fibre-channel@1/hddv@1,0 (hddv1):
      SCSI transport failed: reason 'reset': giving up
      :
      NOTICE: mphd0: I/O path switchover succeeded.
      /pci@80,4000/fibre-channel@1/hddv@1,0 => /pci@84,4000/fibre-channel@1/hddv@2,0
      :

      In the following explanation we assume that above messages were printed out on the console.

   2. Locate the system board numbers for the path interfaces connected the faulty PCI cards with I/O devices in step 1.a (fjpfca3) and display a list of devices under the system boards.
      The system board number is the figure in the below red character part: XY (X represents the cabinet number, Y represents the system board slot number in the cabinet).
      Also, it is understood that the hme0 of network device ("/pci@83,4000/network@1,1" 0 "hme") excluding file devices (hddv1,hddv2,..) exists on the relevant system board from the results of execution of the second command.
      It is possible to ignore about se driver and scf driver (FJSVscfx) with the redundant configuration automatically.
      Refer to "Enhanced Support Facility User's Guide" for details of se driver and scf driver.

      # /opt/FJSVhwr/sbin/drcstat -device | grep /pci@80,4000/fibre-channel@1 <Return>
      00-PCI#0B "/pci@80,4000/fibre-channel@1" 3 "fjpfca"
      00-PCI#0B "/pci@80,4000/fibre-channel@1/hddv@1,0" 1 "hddv"
      00-PCI#0B "/pci@80,4000/fibre-channel@1/hddv@1,1" 2 "hddv"
      :
      # /opt/FJSVhwr/sbin/drcstat -device sb00 <Return>
      00-PCI#0B "/pci@80,4000/fibre-channel@1" 3 "fjpfca"
      00-PCI#0B "/pci@80,4000/fibre-channel@1/hddv@1,0" 1 "hddv"
      00-PCI#0B "/pci@80,4000/fibre-channel@1/hddv@1,1" 2 "hddv"
      :
      00-ONBOARD "/pci@83,4000/ebus@1/FJSV,scfc@14,200000" 0 "FJSVscf2"
      00-ONBOARD "/pci@83,4000/ebus@1/FJSV,se@14,400000" 0 "se"
      00-ONBOARD "/pci@83,4000/network@1,1" 0 "hme"

      In the following, the detach procedure is explained by taking hddv1("/pci@80,4000/fibre-channel@1/hddv@1,0" 1 "hddv") as an example.

   3. Find hddv1 with the "offline fail" status from the results of execution of the iompadm command and determine the logical path name corresponding to the faulty path connected hddv1 (the character string from 'c' to 'd' under /dev/rdsk).
      The following example is the results of execution of MPHD and shows that "c3t1d0" is the logical path name corresponding to the faulty path.
      In case of using MPLB, GRMPD or ETERNUS MPD, specify the "mplb" as -c option parameter of the iompadm command.
      In case of updated GRMPD from MPHD, specify the "mphd" as -c option parameter of the iompadm command.

      # /usr/opt/FJSViomp/bin/iompadm -c mphd -p info <Return>
      :
      IOMP: /dev/FJSVmphd/fiomp/adm2
      -> /devices/pseudo/mphd@2:adm
      Element:
      /dev/rdsk/c3t1d0s2 offline fail block "target completed hard reset sequence [GR7104546- 010000-00-00-30] (hddv1)"
      -> /devices/pci@80,4000/fibre-channel@1/hddv@1,0:c,raw
      /dev/rdsk/c2t2d0s2 online active block "good status with active [GR7104546- 010000-01-01-32] (hddv15)"
      -> /devices/pci@84,4000/fibre-channel@1/hddv@2,0:c,raw
      Node:
      /dev/FJSVmphd/rdsk/mphd2s0
      /dev/FJSVmphd/rdsk/mphd2s1
      /dev/FJSVmphd/rdsk/mphd2s2
      /dev/FJSVmphd/rdsk/mphd2s3
      /dev/FJSVmphd/rdsk/mphd2s4
      /dev/FJSVmphd/rdsk/mphd2s5
      /dev/FJSVmphd/rdsk/mphd2s6
      /dev/FJSVmphd/rdsk/mphd2s7
      Function:
      MPmode=false
      AutoPath=true
      Block=true
      NeedSync=false
      :

3. Disconnect from redundant system

   In the case of MPHD/MPLB/GRMPD/ETERNUS MPD, the path is disconnected from redundant system automatically when the system boards are detached.If you use the redundancy software other than MPHD/MPLB/GRMPD/ETERNUS MPD, see the document for each product.

4. Detaching the system boards

   Detach the system boards equipped with the faulty PCI cards by using the drc(1M) command, and check if the status of the system boards is changed to "Unconfigured".
   In addition, check if the status of the all LUNs (Logical Unit Number) under the logical path in step 2.c is changed to "unconfigured disconnected".
   In case of detaching an active path of the active/standby configuration, the status of the standby side path is changed to active automatically.

   # /opt/FJSVhwr/sbin/drc -disconnect sb00 -keep <Return>
   # /opt/FJSVhwr/sbin/drcstat -board sb00 <Return>
   SB BN Status (Detail) PID Next_PID Board_Type CPU(MHz)
   -- -- --------------- --- -------- ---------- --------
   00 0 Unconfigured 00 00 10 300
   # /usr/opt/FJSViomp/bin/iompadm -c mphd info | grep c3t1d <Return>
   /dev/rdsk/c3t1d0s2 unconfigured disconnected unblock "changing parts with power supply charged [GR7104546- 010000-00-00-30] (hddv1)"
   /dev/rdsk/c3t1d1s2 unconfigured disconnected unblock "changing parts with power supply charged [GR7104546- 010000-00-00-30] (hddv2)"
   :

5. Replacing the PCI cards

   Pull out the system boards detached in step 4 and replace the faulty PCI cards to new parts, then mount back the system boards and connect cable to devices. A certified service engineer takes charge of this work.When replacing the fibre channel PCI cards, the steps below are required.

   [for PCI Fibre Channel(PW008FC3U/PW008FC2U/ GP7B8FC1U)]:

   When using any of SAN management function of Systemwalker/StorageMGR, Softek SANView for ETERNUS (except for Vixel) and SP5000 SRM Facility

   No procedure is necessary. Go to step 6.

   When not using SAN management function of the above products

   In case of replacing of the fibre channel cards of the below environment setting, it is necessary to change the setting of the fibre channel switch SN200 series and the disk array device ETERNUS3000/ETERNUS6000/GR700/800 series.

   • Setting up the zoning by WWPN(World Wide Port Name) on SN200 series
   • Using the host zoning (Host Affinity) function of ETERNUS3000/ETERNUS6000/GR700/800 series

     For details, refer to the manual "Zoning User's Guide" and "ETERNUSmgr User Guide/GRmgr User Guide". If you use the Fibre Channel switch or disk array device other than described above, see the document of each product.

     When changing the above setting, WWPNs (Figures of sixteen digits in all) of the replacement parts are needed. The WWPN of the replacement fibre channel card is determined by using eight-digit number printed on the seal of the card on front plate.This figure of eight digit represents the lower eight digit of WWPN by the hexadecimal number. The upper eight digit of WWPN is 10000000 by a fixed hexadecimal number.For instance, when the following seal puts on the replacement fibre channel card, the WWPN of the replacement card is 100000000e244061.

     0e24
     4061

   [ for Fibre Channel Card (PW028FC3*/PW028FC4*/PW028FC5*)]:

   To replace PCI cards with the following configurations, Fibre Channel switch and disk array device need to be reconfigured individually.

   • If zone configuration by WWPN (World Wide Port Name) is done on Fibre Channel switch.
   • If the Host Zoning function of disk array device is used.

     For details, see the document of each product.

     To perform the above reconfiguration, the WWPN(a 16-digit number) of the replacement card is needed. The WWPN of the PCI card can be known from the twelve characters shown on a label on the back of the card. These characters represent the bottom twelve digits of the WWPN in hexadecimal form. The top four digits are fixed to 1000 in hexadecimal form.For example, if the following label is shown on the back of the card, the WWPN of the replacement card is 10000000c9366037.

     IEEE:0000c9366037

     
     

   Note

   When the zoning setting is changed in SN200 series or other Fibre Channel switch, executing I/O requests to other devices sometimes terminate abnormally with temporary errors by the setting changes. I/O requests to disk array devices and the like are recovered normally by retry processes, but backup processes of fibre channel tape devices sometimes terminate abnormally. Please execute the change of the zoning setting after stopping backup processes.

6. Attaching the system boards

   Using the steps below, attach the detached system board.

   1. Attach the system boards equipped with the replaced PCI cards by using the drc(1M) command, and check if the status of the system boards is changed to "Configured".

      # /opt/FJSVhwr/sbin/drc -connect sb00 <Return>
      # /opt/FJSVhwr/sbin/drcstat -board sb00 <Return>
      SB BN Status (Detail) PID Next_PID Board_Type CPU(MHz)
      -- -- --------------- --- -------- ---------- --------
      00 0 Configured 00 00 10 300

   2. When replacing the fibre channel PCI cards, the steps below are required.
      

      [for PCI Fibre Channel (PW008FC3U/PW008FC2U/ GP7B8FC1U) ]:

      When using any of SAN management function of Systemwalker/StorageMGR, Softek SANView for ETERNUS (except for Vixel) and SP5000 SRM Facility

      Refresh current condition on management window.
      SAN (Storage Area Network) management function searchs and indicates replaced PCI card.
      Use "Inherit of Access Path" function on management window.
      This function will adjust switch zoning and host affinity function of disk array device ETERNUS3000/ETERNUS6000/GR700/800 series with new WWPN of replaced PCI card.
      See "Operation guide" or "User's Manual" of each product for details.

      Note

      When the zoning setting is changed in SN200 series, executing I/O requests to other devices sometimes terminate abnormally with temporary errors by the setting changes.
      I/O requests to disk array devices and the like are recovered normally by retry processes, but backup processes of fibre channel tape devices sometimes terminate abnormally.
      Please execute the change of the zoning setting after stopping backup processes.

      Reattach the system boards equipped with the replaced PCI cards with changed zonig setting by using the following commands.

      # /opt/FJSVhwr/sbin/drc -disconnect sb00 -keep <Return>
      # /opt/FJSVhwr/sbin/drc -connect sb00 <Return>
      # /opt/FJSVhwr/sbin/drcstat -board sb00 <Return>
      SB BN Status (Detail) PID Next_PID Board_Type CPU(MHz)
      -- -- --------------- --- -------- ---------- --------
      00 0 Configured 00 00 10 300

      If you don't use SAN management function of the above products

      No procedure is necessary.

      [ for Fibre Channel Card (PW028FC3*/PW028FC4*/PW028FC5*)]:

      No procedure is necessary.

   3. Check if the status of the all LUNs under the logical path in step 1.c is changed to "online active" or "online standby".
      When switching over the active path in step 2, switch back to the original active side automatically.

      # /usr/opt/FJSViomp/bin/iompadm -c mphd info | grep c3t1d <Return>
      /dev/rdsk/c3t1d0s2 online active block "good status with active [GR7104546- 010000-00-00-30] (hddv1)"
      /dev/rdsk/c3t1d1s2 online active block "good status with active [GR7104546- 010000-00-00-30] (hddv2)"
      :

7. Connect to redundant system

   In the case of MPHD/MPLB/GRMPD/ETERNUS MPD, the path is connected to redundant system automatically when the system boards are attached. If you use the redundancy software other than MPHD/MPLB/GRMPD/ETERNUS MPD, see the document for each product.

8. Start the daemons

   If you use the Fibre Channel Card (PW028FC3*/PW028FC4*/PW028FC5*), execute the following.

   # /etc/rc2.d/S99ElxRMSrv start <Return>
   # /etc/rc2.d/S99ElxDiscSrv start <Return>