| Dynamic Reconfiguration User's Guide I/O device edition - for Solaris (TM) Operating System - |
|
Contents
|
| Chapter 3 File Devices | > 3.2 Expansion Procedure | > 3.2.2 Solaris 9 or later |
This section shows the flow of the procedure of PCI cards expansion for new redundant configuration on Solaris 9 or later system by taking the following redundant software products as examples.
In case of using the other redundant software products, refer to the manual for each product.
If you use the Fibre Channel Card (PW028FC3*/PW028FC4*/PW028FC5*), execute the following.
| # /etc/rc0.d/K10ElxRMSrv stop <Return> # /etc/rc0.d/K10ElxDiscSrv stop <Return> |
In case of expanding PCI cards on the existing redundant system boards, choose the system boards of expansion target and detach the system boards according to step 3 to 4 in Section 3.1.1 or step 4 in Section 3.1.2.
This procedure is not required when expanding the new system boards together.
Using the following procedures, expand the PCI cards. A certified service engineer takes charge of this work.
In expansion of new redundant configuration, it is necessary to expand two or more PCI cards on different system boards, then excute from step 3 to 4 every expansion PCI cards. When expanding the fibre channel PCI cards, the steps below are required.
[for PCI Fibre Channel(PW008FC3U/PW008FC2U/ GP7B8FC1U)]:
When using any of SAN management function of Systemwalker/StorageMGR, Softek SANView for ETERNUS (except for Vixel) and SP5000 SRM Facility
No procedure is necessary. Go to step 6.
When not using SAN management function of the above products
In case of replacing of the fibre channel cards of the below environment setting, it is necessary to change the setting of the fibre channel switch SN200 series and the disk array device ETERNUS3000/ETERNUS6000/GR700/800 series.
For details, refer to the manual "Zoning User's Guide" and "ETERNUSmgr User Guide/GRmgr User Guide". If you use the Fibre Channel switch or disk array device other than described above, see the document of each product.
When changing the above setting, WWPNs (Figures of sixteen digits in all) of the replacement parts are needed. The WWPN of the expansion fibre channel card is determined by using eight-digit number printed on the seal of the card on front plate.
This figure of eight digit represents the lower eight digit of WWPN by the hexadecimal number. The upper eight digit of WWPN is 10000000 by a fixed hexadecimal number.
For instance, when the following seal puts on the expansion fibre channel card, the WWPN of the expansion card is 100000000e244061.
| 0e24 4061 |
[ for Fibre Channel Card (PW028FC3*/PW028FC4*/PW028FC5*)]:
To replace PCI cards with the following configurations, Fibre Channel switch and disk array device need to be reconfigured individually.
For details, see the document of each product.
To perform the above reconfiguration, the WWPN(a 16-digit number) of the replacement card is needed. The WWPN of the PCI card can be known from the twelve characters shown on a label on the back of the card. These characters represent the bottom twelve digits of the WWPN in hexadecimal form. The top four digits are fixed to 1000 in hexadecimal form.
For example, if the following label is shown on the back of the card, the WWPN of the replacement card is 10000000c9366037.
| IEEE:0000c9366037 |
| # /opt/FJSVhwr/sbin/drcstat -device > /tmp/drcstat.pre <Return> |
Note
When the zoning setting is changed in SN200 series or other Fibre Channel switch, executing I/O requests to other devices sometimes terminate abnormally with temporary errors by the setting changes. I/O requests to disk array devices and the like are recovered normally by retry processes, but backup processes of fibre channel tape devices sometimes terminate abnormally. Please execute the change of the zoning setting after stopping backup processes.
Attach the system boards equipped with the expanded PCI cards by using the drc(1M) command, and check if the status of the system boards is changed to "Configured".
In case of detaching the system boards in step 1, restart the stopped high-level products and check the multipath status according to step 7 in Section 3.1.1 or step 6.c in Section 3.1.2.
| # /opt/FJSVhwr/sbin/drc -connect sb00 <Return> # /opt/FJSVhwr/sbin/drcstat -board sb00 <Return> SB BN Status (Detail) PID Next_PID Board_Type CPU(MHz) -- -- --------------- --- -------- ---------- -------- 00 0 Configured 00 00 10 300 |
If you use the Fibre Channel Card (PW028FC3*/PW028FC4*/PW028FC5*), execute the following.
| # /etc/rc2.d/S99ElxRMSrv start <Return> # /etc/rc2.d/S99ElxDiscSrv start <Return> |
When the zoning function is used on fiber channel switch SN200 series configuration, it is necessary to set zoning to the expanded devices.
[for PCI Fibre Channel(PW008FC3U/PW008FC2U/ GP7B8FC1U)]:
When using any of SAN management function of Systemwalker/StorageMGR, Softek SANView for ETERNUS (except for Vixel) and SP5000 SRM Facility
WWPN zoning of SN200 series, Host affinity of Array disk device ETERNUS3000/ETERNUS6000/GR700/800 series and binding setting for fjpfca.conf/sd.conf of host can be added from SAN management window automatically.
See "Operation guide" or "User's Manual" of each product for details.
Before using this function, the following step 7.a.1) and 7.a.2) [except fcp-bind-target setting] are necessary. Moreover, begin from step 7.a.3) after executing this function.
When not using SAN management function of the above products
No procedure is necessary.
[ for Fibre Channel Card (PW028FC3*/PW028FC4*/PW028FC5*)]:
No procedure is necessary.
Note
When the zoning setting is changed in SN200 series, executing I/O requests to other devices sometimes terminate abnormally with temporary errors by the setting changes.
I/O requests to disk array devices and the like are recovered normally by retry processes, but backup processes of fibre channel tape devices sometimes terminate abnormally.
Please execute the change of the zoning setting after stopping backup processes.
Using the steps below, add the setting of drivers.
For file drivers, it is necessary to add the setting of PCI card drivers and device drivers.
If you use ETERNUS MPD, settings of drivers are configured using grmpdautoconf command.
If you use ETERNUS MPD, skip to the procedure "8.Setup of the redundant configuration".
[SCSI drivers (Solaris:glm)]
[PCI Fibre Channel(PW008FC3U/PW008FC2U/ GP7B8FC1U)]
1) Save the results of execution of the drcstat(1M) command in a file, and find a difference between the data in step 3.a and the current result, then determine the instance numbers of fjpfca driver for the expanded two or more PCI cards.
The following example shows that fjpfca1 (instance number: 1) is one of the expanded PCI cards.
Locate the relation between the instance number and the slot position by blinking LED of the expanded PCI card with using the fc_adm command.
Also, check by using the "fc_info -a" command about the relation between the WWPN of expanded card and the instance number.
# /opt/FJSVhwr/sbin/drcstat -device > /tmp/drcstat.pfca <Return>
# diff /tmp/drcstat.pre /tmp/drcstat.pfca | grep fjpfca <Return>
:
> 00-PCI#0B "/pci@80,4000/fibre-channel@1" 1 "fjpfca"
:
# /usr/sbin/FJSVpfca/fc_adm -l fjpfca1 <Return>
2) Edit the setting file "/kernel/drv/fjpfca.conf" and add the below definitions of property for instance numbers in step 7.a.1).port (topology types, link speed)
In case of expanding the PCI cards of fabric connection, define this property.
In FC-AL connection, this definition is not required.fcp-bind-target (binding target ID and WWN)
In case of expanding the disk array devices of fabric connection, define this property.
In FC-AL connection, this definition is not required.max-throttle (the number of maximum SCSI command issues)
This is the property to limit the number of SCSI commands (tag) that can be accepted at the same time.
The necessity and setting value of this property depend on the specification of the disk array devices, so refer to the manual of each disk array device.
The following example is the setting addition of the PCI card for fjpfca1 (instance number: 1) and the disk array device of target=3.
# vi /kernel/drv/fjpfca.conf <Return>
:
port=
"fjpfca0:nport",
"fjpfca1:nport:sp1"; <------ added
fcp-bind-target=
"fjpfca0t0:0x50000e1000002501",
"fjpfca1t3:0x50000e1000002ab0"; <------ added
max-throttle=
"fjpfca0t0:128",
"fjpfca1t3:240"; <------ added
:
3) Notify the fjpfca driver of the attachment of the expansion parts by using the following command.
# /usr/sbin/FJSVpfca/fc_config -v <Return>
[ Fibre Channel driver (PW028FC3*/PW028FC4*/PW028FC5*) ]
Configuration is not required.Compare the output of prtpicl(1M) command with the output saved at 3.a and find the instance number of the added card.
In the following, the procedure is explained by taking the expanded the disk array device with redundant controller configuration (target=3 lun=0 to 2) in step 6 as an example.
[ If you use Solaris:sd ]
1) Ensure target=3,lun=0 target=3,lun=1 target=3,lun=3 are defined in /kernel/drv/sd.conf.If they are not defined or commented, modify /kernel/drv/sd.conf.
# vi /kernel/drv/sd.conf<Return>
name="sd" class="scsi"
target=0 lun=0;
:
name="sd" class="scsi" <------ If undefined, add this definition
target=3 lun=0; <------ or remove the comment symbols.
name="sd" class="scsi"
target=3 lun=1;
name="sd" class="scsi"
target=3 lun=2;
:
2) Notify the sd driver of the attachment of the expanded disk devices by using the following command.
The following messages come out this time. There is no problem.
- # update_drv -f sd <Return>
Cannot unload module: sd
Will be unloaded upon reboot.
Forcing update of sd.conf.
3) Save the results of execution of the drcstat(1M) command in a file, and find a difference between the data in step 3.a and the current result, then determine the physical path names and the logical path names under /dev/dsk of the expanded disk device.
The following example shows that "c3t3d0" and "c4t3d0" are the logical path names of the expanded lun=0. Then the logical path names of lun=1 to 2 are "c3t3d1", "c4t3d1","c3t3d2" and "c4t3d2" (cXtYdZ; X:PCI card number, Y:target ID, Z:lun number).
# prtpicl -v > /tmp/prtpicl.post <Return>
# diff /tmp/prtpicl.pre /tmp/prtpicl.post | grep devfs-path | grep hddv <Return>
> :devfs-path /pci@8d,2000/fibre-channel@1/sd@3,0
> :devfs-path /pci@8d,2000/fibre-channel@1/sd@3,1
> :devfs-path /pci@8d,2000/fibre-channel@1/sd@3,2
> :devfs-path /pci@89,2000/fibre-channel@1/sd@3,0
> :devfs-path /pci@89,2000/fibre-channel@1/sd@3,1
> :devfs-path /pci@89,2000/fibre-channel@1/sd@3,2
# ls -l /dev/dsk | grep /pci@8d,2000/fibre-channel@1/sd@3,0 <Return>
lrwxrwxrwx 1 root root 41 Sep 20 22:53 c3t3d0s0
-> ../../devices/pci@8d,2000/fibre-channel@1/sd@3,0:a
lrwxrwxrwx 1 root root 41 Sep 20 22:53 c3t3d0s1
-> ../../devices/pci@8d,2000/fibre-channel@1/sd@3,0:b
lrwxrwxrwx 1 root root 41 Sep 20 22:53 c3t3d0s2
-> ../../devices/pci@8d,2000/fibre-channel@1/sd@3,0:c
lrwxrwxrwx 1 root root 41 Sep 20 22:53 c3t3d0s3
-> ../../devices/pci@8d,2000/fibre-channel@1/sd@3,0:d
lrwxrwxrwx 1 root root 41 Sep 20 22:53 c3t3d0s4
-> ../../devices/pci@8d,2000/fibre-channel@1/sd@3,0:e
lrwxrwxrwx 1 root root 41 Sep 20 22:53 c3t3d0s5
-> ../../devices/pci@8d,2000/fibre-channel@1/sd@3,0:f
lrwxrwxrwx 1 root root 41 Sep 20 22:53 c3t3d0s6
-> ../../devices/pci@8d,2000/fibre-channel@1/sd@3,0:g
lrwxrwxrwx 1 root root 41 Sep 20 22:53 c3t3d0s7
-> ../../devices/pci@8d,2000/fibre-channel@1/sd@3,0:h
# ls -l /dev/dsk | grep /pci@89,2000/fibre-channel@1/sd@3,0 <Return>
lrwxrwxrwx 1 root root 41 Sep 20 22:53 c4t3d0s0
-> ../../devices/pci@89,2000/fibre-channel@1/sd@3,0:a
:
3) Label a disk and perform the setting of partitions (slices) for the logical path names in step 7.b.3) by using the format(1M) command.
[ Disk array device (hard disk driver : hddv) ]
1) Check the presence of the file "/kernel/drv/hddv.conf".
If it does not exist, this case is "Hard Disk Driver first installation", so go to step 7.b.3), and if it exists, this case is "non-hot expansion settings", so go to step 5.b.2).
- [In case of not existing (first installation)] --> go to step 7.b.3)
# ls /kernel/drv/hddv.conf <Return>
/kernel/drv/hddv.conf: No such file or directory
- [In case of existing (non-hot expansion settings)] --> go to step 7.b.2)
# ls /kernel/drv/hddv.conf <Return>
/kernel/drv/hddv.conf
2) After executing the following command on non-hot expansion specified in the settings, check if the file "/kernel/drv/hddv.conf" is deleted.
# hddvadm -i <Return>
# ls /kernel/drv/hddv.conf <Return>
/kernel/drv/hddv.conf: No such file or directory
3) Using the following command, create the hddv.conf file for hot expansion function.
The message example in "Non-hot expansion settings" shows disk array device GR720 of target=1(lun=0) with using now is overlapped in sd/hddv drivers.
- [First installation] --> go to step 7.b.5)
# hddvadm -M <Return>
- [Non-hot expansion settings] --> go to step 7.b.4)
# hddvadm -M <Return>
already ID= 1 exists. /devices/pci@1d,2000/fibre-channel@1/sd@1,0:a,raw : GR720
4) When the hot expansion function is used, the target IDs registered in hddv.conf cannot be registered in sd.conf.
So if definition in sd.conf and hddv.conf are overlapped, delete or comment out the duplicated definitions of sd.conf
The below example shows the duplicated definition of target=1.
# vi /kernel/drv/sd.conf <Return>
:
name="sd" class="scsi"
target=0 lun=0;
:
#name="sd" class="scsi" <------
target=1 lun=0;
#name="sd" class="scsi" Delete or comment out
target=1 lun=1; the duplicated definitions of sd and hddv drivers.
#name="sd" class="scsi"
target=1 lun=2; <------
:
But it is impossible to delete the target IDs being recognized as a system volume a CD-ROM device etc. in sd driver.
In this case, change the target ID on hddv.conf and the disk array device side to unduplicated definition of sd.conf.
5) Edit the file "/kernel/drv/hddv.conf" generated automatically by using the hddvadm -M command, remove the comment symbols of the LUN definitions of hot expansion and using in non-hot expansion settings.
When the target number of each disk array device controller for redundant configuration is different, add each target number definition at the same time.
If the definition of LUNs that are expected to expand is found, because of second or later execution of hot expansion, it is not required to edit.
The below example shows registrations of hot expansion targets (target=3, lun=0 to 2) and disk array device GR720 using in non-hot expansion settings (target=1)
# ls /kernel/drv/hddv.conf <Return>
/kernel/drv/hddv.conf
# vi /kernel/drv/hddv.conf <Return>
:
#name="hddv" class="scsi" target=0 lun=0;
:
#name="hddv" class="scsi" target=0 lun=7;
:
name="hddv" class="scsi" target=1 lun=0; <------
name="hddv" class="scsi" target=1 lun=1; Remove the comment symbols of using LUNs
name="hddv" class="scsi" target=1 lun=2; <------
:
name="hddv" class="scsi" target=3 lun=0; <------
name="hddv" class="scsi" target=3 lun=1; Remove the comment symbols of
name="hddv" class="scsi" target=3 lun=2; <------ expanded LUNs
#name="hddv" class="scsi" target=3 lun=3;
:
6) Check if no duplicate definitions exist in the sd.conf and hddv.conf files by using the hddvchk_conf command.
The message example in "Check NG" shows the duplicated definition is target=3, so go back to step 5.b.4).
Also, the following example is execution when the FJSVhddv package is installed to /opt.
The placed path of the hddvchk_conf command changes by installed directory.
- [Check OK] --> go to step 7.b.7)
# /opt/FJSVhddv/bin/hddvchk_conf <Return>
/kernel/drv/sd.conf and /kernel/drv/hddv.conf seems to be correct
- [Check NG] --> go back to step 7.b.4)
# /opt/FJSVhddv/bin/hddvchk_conf <Return>
NG ID = [ 3 ]
7) Notify the hard disk driver of the attachment of the expanded LUNs by using the following command.
# update_drv -f hddv <Return>
Under the present using disk array devices, the following messages come out. There is no problem.
# update_drv -f hddv <Return>
Cannot unload module: hddv
Will be unloaded upon reboot.
Forcing update of hddv.conf.
8) Save the results of execution of the drcstat(1M) command in a file, and find a difference between the data in step 3.a and the current result, then determine the physical path names and the logical path names under /dev/dsk of the expanded LUNs.
The following example shows that "c3t3d0" and "c4t3d0" are the logical path names of the expanded lun=0. Then the logical path names of lun=1 to 2 are "c3t3d1", "c4t3d1","c3t3d2" and "c4t3d2" (cXtYdZ; X:PCI card number, Y:target ID, Z:lun number).
# /opt/FJSVhwr/sbin/drcstat -device > /tmp/drcstat.post <Return>
# diff /tmp/drcstat.pre /tmp/drcstat.post | grep hddv <Return>
:
> 00-PCI#0B "/pci@80,4000/fibre-channel@1/hddv@3,0" 10 "hddv"
> 00-PCI#0B "/pci@80,4000/fibre-channel@1/hddv@3,1" 11 "hddv"
> 00-PCI#0B "/pci@80,4000/fibre-channel@1/hddv@3,2" 12 "hddv"
> 03-PCI#1B "/pci@8d,4000/fibre-channel@1/hddv@3,0" 13 "hddv"
> 03-PCI#1B "/pci@8d,4000/fibre-channel@1/hddv@3,1" 14 "hddv"
> 03-PCI#1B "/pci@8d,4000/fibre-channel@1/hddv@3,2" 15 "hddv"
# ls -l /dev/dsk | grep /pci@80,4000/fibre-channel@1/hddv@3,0 <Return>
lrwxrwxrwx 1 root root 41 Sep 20 22:53 c3t3d0s0
-> ../../devices/pci@80,4000/fibre-channel@1/hddv@3,0:a
lrwxrwxrwx 1 root root 41 Sep 20 22:53 c3t3d0s1
-> ../../devices/pci@80,4000/fibre-channel@1/hddv@3,0:b
lrwxrwxrwx 1 root root 41 Sep 20 22:53 c3t3d0s2
-> ../../devices/pci@80,4000/fibre-channel@1/hddv@3,0:c
lrwxrwxrwx 1 root root 41 Sep 20 22:53 c3t3d0s3
-> ../../devices/pci@80,4000/fibre-channel@1/hddv@3,0:d
lrwxrwxrwx 1 root root 41 Sep 20 22:53 c3t3d0s4
-> ../../devices/pci@80,4000/fibre-channel@1/hddv@3,0:e
lrwxrwxrwx 1 root root 41 Sep 20 22:53 c3t3d0s5
-> ../../devices/pci@80,4000/fibre-channel@1/hddv@3,0:f
lrwxrwxrwx 1 root root 41 Sep 20 22:53 c3t3d0s6
-> ../../devices/pci@80,4000/fibre-channel@1/hddv@3,0:g
lrwxrwxrwx 1 root root 41 Sep 20 22:53 c3t3d0s7
-> ../../devices/pci@80,4000/fibre-channel@1/hddv@3,0:h
# ls -l /dev/dsk | grep /pci@8d,4000/fibre-channel@1/hddv@3,0 <Return>
lrwxrwxrwx 1 root root 41 Sep 20 22:53 c4t3d0s0
-> ../../devices/pci@8d,4000/fibre-channel@1/hddv@3,0:a
:
9) Label a disk and perform the setting of partitions (slices) for the logical path names in step 7.b.8) by using the format(1M) command.
It is sufficient to execute for one representative logical path name of each expanded LUN, so need not to execute for every all PCI cards.
[ SCSI driver(Solaris: glm , FUJITSU PCI SCSI Adapter Driver) ]
No procedure is necessary.
[ Fibre Channel driver (PW008FC3U/PW008FC2U/ GP7B8FC1U) ]
No procedure is necessary.
[ Fibre Channel Card (PW028FC3*/PW028FC4*/PW028FC5*)]
Bind the auto mapped target connected with the card and modify the entries of lpfc.conf to match the requirements of the extended Fibre Channel Controller. These procedures are reflected on next boot time. In the following, indicate to bind the auto mapped target with the card.
| # /usr/sbin/lpfc/lputil <Return> LightPulse Common Utility for Solaris/SPARC. Version 1.6a8 (03/03/2004). Copyright (c) 2003, Emulex Network Systems, Inc. Emulex Fibre Channel Host Adapters Detected: 2 Host Adapter 0 (lpfc11) is an LP9802 (Ready Mode) Host Adapter 1 (lpfc9) is an LP9802 (Ready Mode) MAIN MENU 1. List Adapters 2. Adapter Information 3. Firmware Maintenance 4. Reset Adapter 5. Persistent Bindings 0. Exit Enter choice => 5 <Return> PERSISTENT BINDINGS MENU 1. Display Current Bindings 2. Display All Nodes 3. Duplicate Adapter Bindings 4. Bind Target Manually 5. Bind Automapped Targets 6. Delete Binding(s) 0. Return to Main Menu Enter choice => 5 <Return> 0. lpfc11 1. lpfc9 Select an adapter => 0<Return> Automapped FCP Nodes: Node Target WWPN WWNN ---- ------ --------- -------- Automapped FCP Node 3 50-06-0e-80-00-10-04-92 50-06-0e-80-00-10-04-92 Press the Enter or Return key to continue: Bind all auto-mapped targets? (Y/N) => Y <Return> Select Binding Method: 1. By Node Name 2. By Port Name 3. By D_ID 0. Cancel Enter Choice => 2 <Return> |
Set up the new redundant configuration by using the following command.
In case of using MPHD or MPLB2.1, specify the logical path names in step 7.b.8) as the command parameter, and need to execute the command every expanded LUNs.
The below red character strings are the new configured path names of redundant software, and the blue character strings are the each logical path names configuring the path of redundant software. The following procedure is described for MPHD/MPLB/GRMPD/ETERNUS MPD. If you use the redundancy software other than MPHD/MPLB/GRMPD/ETERNUS MPD, see the document for each product.
|
Add the setting of the high-level products (services) by using the path names of redundant software in step 8.
[File system operation]
| # newfs /dev/FJSVmphd/rdsk/mphd2s6 <Return> # vi /etc/vfstab <Return> : /dev/FJSVmphd/dsk/mphd2s6 /dev/FJSVmphd/rdsk/mphd2s6 /mnt ufs 1 yes - : |
[Raw access operation]
[Swap device]
| # swap -a /dev/FJSVmphd/dsk/mphd2s4 <Return> # vi /etc/vfstab <Return> : /dev/FJSVmphd/dsk/mphd2s4 - - swap - no - : # swap -l <Return> swapfile dev swaplo blocks free : /dev/FJSVmphd/dsk/mphd2s4 154,132 16 788384 683680 |
Start the high-level products (services) by using the path names of redundant software in step 8.
[File system operation]
| # mount -F ufs /dev/FJSVmphd/dsk/mphd2s6 /mnt <Return> |
[Raw access operation]
[Swap device]
|
Contents
|