| PCI Hot Plug User's Guide I/O device edition - for Solaris (TM) Operating Environment - |
|
Contents
|
| Chapter 3 File Devices |
This section describes the procedures to exchange PCI cards using the PCI Hot Plug feature on non-redundant and redundant systems.
Before exchanging PCI cards without using redundancy software such as multipath control, applications using the PCI cards must be stopped.
Follow the procedures below to exchange PCI cards on a non-redundant system.
Execute the following command to stop the hardware monitoring daemon of machine administration.
| # /usr/sbin/FJSVmadm/prephp <Return> |
If you use the Fibre Channel Card (PW028FC3*/PW028FC4*/PW028FC5*), execute the following. The daemons will be stopped.
| # /etc/rc0.d/K10ElxRMSrv stop <Return> # /etc/rc0.d/K10ElxDiscSrv stop <Return> |
Follow the instructions below and determine the interface name of the path connecting the target PCI card and I/O devices and the connected I/O device.
In the example below, glm2 is the interface name of the path connecting the broken PCI card and I/O devices, and sd20 is the disk device connected to glm2.
When exchanging a PCI card which has two ports, the physical path name of the connection path between the I/O device corresponding to the other port, and the I/O device connected to that path must be determined in the same manner.
The physical path name corresponding to the other port can be determined by the fact that the name of the two ports on the same PCI card have a relation like "/pci@89,4000/scsi@2" and "/pci@89,4000/scsi@2,1." The I/O device connected to that path resides under the directory of the physical path name determined.
| : WARNING: /pci@89,4000/scsi@2 (glm2): invalid intcode=fe00 : WARNING: /pci@89,4000/scsi@2/sd@3,0 (sd20): SCSI transport failed: reason 'reset': giving up : |
If you use the Fibre Channel Card (PW028FC3*/PW028FC4*/PW028FC5*), the messages are shown as follows.
| : WARNING: lpfc0:INe:Adapter Hardware Error : WARNING: /pci@89,4000/fibre-channel@2/sd@3,0 (sd20): SCSI transport failed: reason 'reset': giving up |
Exec the following command and check the physical path name.
| # grep lpfc /etc/path_to_inst <Return> "/pci@89,4000/fibre-channel@2" 0 "lpfc" |
The second value is instance number of each lpfc instance.
| # ls -l /dev/dsk | grep /pci@89,4000/scsi@2/sd@3,0 <Return> lrwxrwxrwx 1 root root 41 Sep 20 22:53 c2t3d0s0 -> ../../devices/pci@89,4000/scsi@2/sd@3,0:a lrwxrwxrwx 1 root root 41 Sep 20 22:53 c2t3d0s1 -> ../../devices/pci@89,4000/scsi@2/sd@3,0:b lrwxrwxrwx 1 root root 41 Sep 20 22:53 c2t3d0s2 -> ../../devices/pci@89,4000/scsi@2/sd@3,0:c lrwxrwxrwx 1 root root 41 Sep 20 22:53 c2t3d0s3 -> ../../devices/pci@89,4000/scsi@2/sd@3,0:d lrwxrwxrwx 1 root root 41 Sep 20 22:53 c2t3d0s4 -> ../../devices/pci@89,4000/scsi@2/sd@3,0:e lrwxrwxrwx 1 root root 41 Sep 20 22:53 c2t3d0s5 -> ../../devices/pci@89,4000/scsi@2/sd@3,0:f lrwxrwxrwx 1 root root 41 Sep 20 22:53 c2t3d0s6 -> ../../devices/pci@89,4000/scsi@2/sd@3,0:g lrwxrwxrwx 1 root root 41 Sep 20 22:53 c2t3d0s7 -> ../../devices/pci@89,4000/scsi@2/sd@3,0:h |
Stop applications with the following operations.
| # sh /etc/init.d/volmgt stop <Return> |
[ If the device is a disk unit (file system operation) ]
| # mount | grep c2t3d0 <Return> /export/home on /dev/dsk/c2t3d0s3 setuid/read/write/largefiles on Mon Sep 30 01:00:51 2002 /develop/firm on /dev/dsk/c2t3d0s0 setuid/read/write/largefiles on Mon Sep 30 01:00:51 2002 /develop/drv on /dev/dsk/c2t3d0s1 setuid/read/write/largefiles on Mon Sep 30 01:00:51 2002 /pub on /dev/dsk/c2t3d0s6 setuid/read/write/largefiles on Mon Sep 30 01:00:50 2002 |
| # fuser -c /export/home <Return> /export/home: 14967c 14571c 14493ctm 14020c 13828tm 13803c 13575c 13133c 13125tm 13107c 12682ctm 12066tm 12048c 11971ctm 11952ctm 11937c 11867c 11846c 349m |
| # umount /export/home <Return> # umount /develop/firm <Return> # umount /develop/drv <Return> # umount /pub <Return> |
[ If the device is a disk unit (raw access operation) or a tape device ]
| # iostat -xc <Return> extended device statistics cpu device r/s w/s kr/s kw/s wait actv svc_t %w %b us sy wt id sd0 59.7 7.5 474.5 45.0 0.0 3.9 58.6 0 41 3 7 23 67 sd1 0.1 0.3 1.0 2.5 0.0 0.0 16.0 0 0 sd20 0.0 0.1 0.3 0.7 0.0 0.0 14.7 0 0 st82 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 nfs1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 |
Disk access statistics of the target disk can be checked as below.
It is the same with tape devices. The example below shows access statistics of one minute.
| # sar -d 60 1 <Return> SunOS machine0 5.8 Generic_108528-05 sun4u 10/02/02 17:56:00 device %busy avque r+w/s blks/s avwait avserv 17:57:00 nfs1 0 0.0 0 0 0.0 0.0 sd0 2 0.3 2 37 0.0 145.5 sd0,a 1 0.1 0 5 0.0 301.4 sd0,b 0 0.0 0 9 0.0 31.5 sd0,c 0 0.0 0 0 0.0 0.0 sd0,d 1 0.0 0 4 0.0 126.8 sd0,e 0 0.0 0 0 0.0 0.0 sd0,f 1 0.1 1 6 0.0 120.9 sd0,g 1 0.1 1 14 0.0 111.7 : sd20 0 0.0 0 0 0.0 0.0 sd20,a 0 0.0 0 0 0.0 0.0 sd20,c 0 0.0 0 0 0.0 0.0 sd20,g 0 0.0 0 0 0.0 0.0 : st82 0 0.0 0 0 0.0 0.0 : |
[ If the device is a disk unit (swap device) ]
| # swap -l <Return> swapfile dev swaplo blocks free /dev/dsk/c2t3d0s4 32,164 16 788384 683680 |
| # swap -d /dev/dsk/c2t3d0s4 <Return> |
Disconnect the target PCI card with the following procedure.
Please refer to "PCI Hot Plug User's Guide" about the detail of inst2comp(1M) command. In this example, the "Ap_Id" is "pcipsy3:C0M00-PCI#slot02"
| # /usr/sbin/FJSVmadm/inst2comp glm2 <Return> pcipsy3:C0M00-PCI#slot02 |
| # cfgadm pcipsy3:C0M00-PCI#slot02 <Return> Ap_Id Type Receptacle Occupant Condition pcipsy3:C0M00-PCI#slot02 mult/hp connected configured ok |
| # cfgadm -c disconnect pcipsy3:C0M00-PCI#slot02 <Return> # cfgadm pcipsy3:C0M00-PCI#slot02 <Return> Ap_Id Type Receptacle Occupant Condition pcipsy3:C0M00-PCI#slot02 unknown disconnected unconfigured unknown |
Note:
When error occurs during disconnect, cfgadm command unusually fail with following message. If cfgadm command fails, execute the command once again.
cfgadm: Component system is busy, try again: disconnect failed
| # cfgadm -x led=fault,mode=blink pcipsy3:C0M00-PCI#slot02 <Return> |
Replace the PCI card disconnected in 4. with a replacement card and connect cable to devices. This operation is performed by our customer support.
When exchanging Fibre Channel cards, the following operations are also required.
[ for PCI Fibre Channel(PW008FC3U/PW008FC2U/ GP7B8FC1U)]:
No procedure is necessary. Go to step 7.
To replace PCI cards with the following configurations, Fibre Channel switch SN200 series and disk array device ETERNUS3000/ETERNUS6000/GR700/800 series need to be reconfigured individually.
- If zone configuration by WWPN(World Wide Port Name) is done on SN200 series.
- If the Host Affinity function of ETERNUS3000/ETERNUS6000/GR700/800 series is used.
For details, see "SN200 Series Affinity User's Guide" or "ETERNUS3000/ETERNUS6000/GR700/800 series GRmgr User's Guide." If you use the Fibre Channel switch or disk array device other than described above, see the document of each product.
To perform the above reconfiguration, the WWPN(a 16-digit number) of the replacement card is needed. The WWPN of the PCI card can be known from the eight characters shown on a label on the front plate of the card. These characters represent the bottom eight digit of the WWPN in hexadecimal form. The top eight digit are fixed to 10000000 in hexadecimal form.
For example, if the following label is shown on the front plate of the card, the WWPN of the replacement card is 100000000e244061.
| 0e244061 |
[ for Fibre Channel Card (PW028FC3*/PW028FC4*/PW028FC5*)]:
To replace PCI cards with the following configurations,
Fibre Channel switch and disk array device need to be reconfigured individually.
For details, see the document of each product.
To perform the above reconfiguration, the WWPN(a 16-digit number) of the replacement card is needed. The WWPN of the PCI card can be known from the twelve characters shown on a label on the back of the card. These characters represent the bottom twelve digit of the WWPN in hexadecimal form. The top four digit are fixed to 1000 in hexadecimal form.
For example, if the following label is shown on the back of the card, the WWPN of the replacement card is 10000000c9366037.
| IEEE:0000c9366037 |
Note:
When changing Affinity configuration on SN200 series or other Fibre Channel switch, I/O to other devices is effected by the change, and may result in temporal errors.
I/O to disk array devices recovers normally because of retry processes, but on Fibre Channel tape devices, backup processes may end in errors. Stop backup before changing Affinity configuration
.
Connect the replaced PCI card using the cfgadm(1M) command with the configure option, or by pushing the button corresponding to the replacement slot position. Note that the push button is only effective in multiuser mode. After the new PCI card is connected, use the cfgadm(1M) command and confirm that the slot status has changed to "connected configured."
If a large-scale configuration of I/O devices is connected to the PCI card in the target slot, command execution for status confirmation may take time.
| # cfgadm -c configure pcipsy3:C0M00-PCI#slot02 <Return> # cfgadm pcipsy3:C0M00-PCI#slot02 <Return> Ap_Id Type Receptacle Occupant Condition pcipsy3:C0M00-PCI#slot02 mult/hp connected configured ok |
When exchanging Fibre Channel cards, the following operations are also required.
[ for PCI Fibre Channel(PW008FC3U/PW008FC2U/ GP7B8FC1U)]:
SAN (Storage Area Network) management function searches and indicates replaced PCI card. This function will adjust switch zoning and host affinity function of disk array device ETERNUS3000/ETERNUS6000/GR700/800 series with new WWN of replaced PCI card. See "Operation guide" or "User's Manual" of each product for details.
Note:
When changing Affinity configuration on SN200 series, I/O to other devices is effected by the change, and may result in temporal errors.
I/O to disk array devices recovers normally because of retry processes, but on Fibre Channel tape devices, backup processes may end in errors. Stop backup before changing Affinity configuration.
| # cfgadm -c disconnect pcipsy3:C0M00-PCI#slot02 <Return> # cfgadm -c configure pcipsy3:C0M00-PCI#slot02 <Return> # cfgadm pcipsy3:C0M00-PCI#slot02 <Return> Ap_Id Type Receptacle Occupant Condition pcipsy3:C0M00-PCI#slot02 mult/hp connected configured ok |
[ for Fibre Channel Card (PW028FC3*/PW028FC4*/PW028FC5*)]:
No procedure is necessary. Go to step 8.
Restart the stopped applications with the following operations.
| # sh /etc/init.d/volmgt start <Return> |
[ If the device is a disk unit (file system operation) ]
Mount the unmounted filesystem, and resume usage.
| # mount -F ufs /dev/dsk/c2t3d0s3 /export/home <Return> # mount -F ufs /dev/dsk/c2t3d0s0 /develop/firm <Return> # mount -F ufs /dev/dsk/c2t3d0s1 /develop/drv <Return> # mount -F ufs /dev/dsk/c2t3d0s6 /pub <Return> |
[ If the device is a disk unit (raw access operation) or a tape device ]
Restart applications and resume usage.
Refer to the manual of each application for details.
[ If the device is a disk unit (swap device) ]
Add swap device, and resume usage.
| # /sbin/swapadd -2 <Return> |
The daemons will be started.
| # /etc/rc2.d/S99ElxRMSrv start <Return> # /etc/rc2.d/S99ElxDiscSrv start <Return> |
Execute the following commands to update hardware configuration information of machine administration and to restart the hardware monitoring daemon.
| # /usr/sbin/FJSVmadm/postphp <Return> |
PCI cards can be replaced without stopping services such as user applications on a redundant system using software such as multipath control.
Here, the procedure to replace PCI cards on a redundant system using the following redundancy software products is explained.
- Multipath Disk Control (MPHD)
- GR Multipath Driver (GRMPD)
If other redundancy software products are used, see the manual of each product.
Use the following command to stop the hardware monitoring daemon of machine administration.
| # /usr/sbin/FJSVmadm/prephp <Return> |
The daemons will be stopped.
| # /etc/rc0.d/K10ElxRMSrv stop <Return> # /etc/rc0.d/K10ElxDiscSrv stop <Return> |
Follow the instructions below and determine the interface name of the path connecting the target PCI card and I/O devices and the connected I/O device. The following procedure is described for MPHD/GRMPHD. If you use the redundancy software other than MPHD/GRMPHD, see the document for each product.
In the example below, fjpfca3 interface name of the path connecting the target PCI card and I/O devices, and hddv1is the LUN(Logical Unit Number) of the disk array device connected to fjpfca3.
| : WARNING: /pci@8d,2000/fibre-channel@1 (fjpfca3): Hard Error : PCI DMA error. : WARNING: /pci@8d,2000/fibre-channel@1/hddv@1,0 (hddv1): SCSI transport failed: reason 'reset': giving up : NOTICE: mphd0: I/O path switchover succeeded. /pci@8d,2000/fibre-channel@1/hddv@1,0 => /pci@89,2000/fibre- channel@1/hddv@2,0 : |
The following procedures are also explained assuming the console messages above.
The following is an example of MPHD, and "/dev/rdsk/c3t1d0s2" is the logical path name corresponding to hddv1.
In case of GRMPD, specify "mplb" in the -c option parameter of the iompadm command. In case of updated GRMPD from MPHD, specify the "mphd" as -c option parameter of the iompadm command.
| # /usr/opt/FJSViomp/bin/iompadm -c mphd -p info <Return> : IOMP: /dev/FJSVmphd/fiomp/adm2 -> /devices/pseudo/mphd@2:adm Element: /dev/rdsk/c3t1d0s2 offline fail block "target completed hard reset sequence [GR7104546- 010000-00-00-30] (hddv1)" -> /devices/pci@8d,2000/fibre-channel@1/hddv@1,0:c,raw /dev/rdsk/c2t2d0s2 online active block "good status with active [GR7104546- 010000-01-01-32] (hddv15)" -> /devices/pci@89,2000/fibre-channel@1/hddv@2,0:c,raw Node: /dev/FJSVmphd/rdsk/mphd2s0 /dev/FJSVmphd/rdsk/mphd2s1 /dev/FJSVmphd/rdsk/mphd2s2 /dev/FJSVmphd/rdsk/mphd2s3 /dev/FJSVmphd/rdsk/mphd2s4 /dev/FJSVmphd/rdsk/mphd2s5 /dev/FJSVmphd/rdsk/mphd2s6 /dev/FJSVmphd/rdsk/mphd2s7 Function: MPmode=false AutoPath=true Block=true NeedSync=false : |
Disconnect the path between the target PCI card and I/O devices.
The following procedure is described for MPHD/GRMPHD. If you use the redundancy software other than MPHD/GRMPD, see the document for each product.
Execute the following command specifying the logical path name of the connected LUN determined in procedure 3.b. This command only needs to be executed on the representation LUN, and does not need to be executed on each LUN under the same PCI card.
If the operation path is disconnected on a redundant system, the standby path will automatically be switched to operation.
If both MPHD/GRMPD controlled devices are connected to the target PCI card, the commands for both products need to be executed.
| MPHD # /usr/opt/FJSViomp/bin/iompadm -c mphd change adapter_disconnect /dev/rdsk/c3t1d0s2 <Return> GRMPD # /usr/opt/FJSViomp/bin/iompadm -c mplb change adapter_disconnect /dev/rdsk/c3t1d0s2 <Return> |
Disconnect the defected PCI card with the following procedures.
| # /usr/sbin/FJSVmadm/inst2comp fjpfca3 <Return> pcipsy21:R0B01-PCI#slot03 |
| # cfgadm pcipsy21:R0B01-PCI#slot03 <Return> Ap_Id Type Receptacle Occupant Condition pcipsy21:R0B01-PCI#slot03 fibre/hp connected configured ok |
| # cfgadm -c disconnect pcipsy21:R0B01-PCI#slot03 <Return> # cfgadm pcipsy21:R0B01-PCI#slot03 <Return> Ap_Id Type Receptacle Occupant Condition pcipsy21: R0B01-PCI#slot03 unknown disconnected unconfigured unknown |
Note:
When error occurs during disconnect, cfgadm command unusually fail with following message. If cfgadm command fails, execute the command once again.
cfgadm: Component system is busy, try again: disconnect failed
| # cfgadm -x led=fault,mode=blink pcipsy21:R0B01-PCI#slot03 <Return> |
Replace the PCI card disconnected in 4. with a replacement card and connect cable to devices. This operation is performed by our customer support.
When exchanging Fibre Channel cards, the following operations are also required.
[ for PCI Fibre Channel(PW008FC3U/PW008FC2U/ GP7B8FC1U)]:
If you use SAN management function of Systemwalker StorageMGR/Softek SANView for ETERNUS (except for Vixel) /SP5000 SRM Facility
No procedure is necessary. Go to step 7.
If you don't use SAN management function of the above products
To replace PCI cards with the following configurations, Fibre Channel switch SN200 series and disk array device ETERNUS3000/ETERNUS6000/GR700/800 series need to be reconfigured individually.
- If zone configuration by WWPN(World Wide Port Name) is done on SN200 series.
- If the Host Affinity function of ETERNUS3000/ETERNUS6000/GR700/800 series is used.
For details, see "SN200 Series Affinity User's Guide" or "ETERNUS3000/ETERNUS6000/GR700/800 series GRmgr User's Guide." If you use the Fibre Channel switch or disk array device other than described above, see the document of each product.
To perform the above reconfiguration, the WWPN(a 16-digit number) of the replacement card is needed. The WWPN of the PCI card can be known from the eight characters shown on a label on the front plate of the card. These characters represent the bottom eight digit of the WWPN in hexadecimal form. The top eight digit are fixed to 10000000 in hexadecimal form.
For example, if the following label is shown on the front plate of the card, the WWPN of the replacement card is 100000000e244061.
| 0e24 4061 |
[ for Fibre Channel Card (PW028FC3*/PW028FC4*/PW028FC5*)]:
To replace PCI cards with the following configurations, Fibre Channel switch and disk array device need to be reconfigured individually.
- If zone configuration by WWPN (World Wide Port Name) is done on Fibre Channel switch.
- If the Host Zoning function of disk array device is used.
For details, see the document of each product.
To perform the above reconfiguration, the WWPN(a 16-digit number) of the replacement card is needed. The WWPN of the PCI card can be known from the twelve characters shown on a label on the back of the card. These characters represent the bottom twelve digit of the WWPN in hexadecimal form. The top four digit are fixed to 1000 in hexadecimal form.
For example, if the following label is shown on the back of the card, the WWPN of the replacement card is 10000000c9366037.
| IEEE:0000c9366037 |
Note:
When changing Affinity configuration on SN200 series or other Fibre Channel switch, I/O to other devices is effected by the change, and may result in temporal errors.
I/O to disk array devices recovers normally because of retry processes, but on Fibre Channel tape devices, backup processes may end in errors. Stop backup before changing Affinity configuration.
Connect the replaced PCI card using the cfgadm(1M) command with the configure option, or by pushing the button corresponding to the replacement slot position. Note that the push button is only effective in multiuser mode. After the new PCI card is connected, use the cfgadm(1M) command and confirm that the slot status has changed to "connected configured."
If a large-scale configuration of I/O devices is connected to the PCI card in the target slot, command execution for status confirmation may take time.
| # cfgadm -c configure pcipsy3:C0M00-PCI#slot02 <Return> # cfgadm pcipsy3:C0M00-PCI#slot02 <Return> Ap_Id Type Receptacle Occupant Condition pcipsy3:C0M00-PCI#slot02 mult/hp connected configured ok |
When exchanging Fibre Channel cards, the following operations are also required.
[ for PCI Fibre Channel(PW008FC3U/PW008FC2U/ GP7B8FC1U)]:
| # cfgadm -c disconnect pcipsy3:C0M00-PCI#slot02 <Return> # cfgadm -c configure pcipsy3:C0M00-PCI#slot02 <Return> # cfgadm pcipsy3:C0M00-PCI#slot02 <Return> Ap_Id Type Receptacle Occupant Condition pcipsy3:C0M00-PCI#slot02 mult/hp connected configured ok |
- If you don't use SAN management function of the above products
No procedure is necessary. Go to step 7.
[ for Fibre Channel Card (PW028FC3*/PW028FC4*/PW028FC5*)]:
No procedure is necessary. Go to step 7.
The following procedure is described for MPHD/GRMPHD. If you use the redundancy software other than MPHD/GRMPHD, see the document for each product.
Execute the following command specifying the logical path name from 3.b. If the operating path switched in procedure 3., the operating path will be automatically switched back.
If both MPHD/GRMPD controlled devices are connected to the PCI card to replace, the commands for both products need to be executed.
| MPHD # /usr/opt/FJSViomp/bin/iompadm -c mphd restart adapter_connect /dev/rdsk/c3t1d0s2 <Return> GRMPD # /usr/opt/FJSViomp/bin/iompadm -c mplb restart adapter_connect /dev/rdsk/c3t1d0s2 <Return> |
The daemons will be started.
| # /etc/rc2.d/S99ElxRMSrv start <Return> # /etc/rc2.d/S99ElxDiscSrv start <Return> |
Execute the following commands to update hardware configuration information of machine administration and to restart the hardware monitoring daemon.
| # /usr/sbin/FJSVmadm/postphp <Return> |
|
Contents
|