6.7.3 Setting Up Resources

This section describes the procedure for creating Cmdline resources.

For notes when setting Cmdline resources, see "6.11 Notes When Setting Cmdline Resources."

Operation Procedure:

Select "CommandLines" from "turnkey wizard STANDBY."
"CommandLines" will appear. Select "AdditionalStartCommand."
Select "FREECHOICE" and then enter the full path of the StartCommand. If you need to specify arguments, delimit them with blanks.
StartCommand is executed during Online processing to start user applications.
[StartCommand exit codes]
StartCommand has the following exit codes:
0: Normal exit. The Online processing is successfully done.
Other than 0: Abnormal exit. The Online processing fails. When the script exits with the cord other than 0, the resource will enter Faulted.
Note
The following characters cannot be used in the script path and the arguments that set for StartCommand, and StopCommand and CheckCommand to be described later.
```
    =    \    ~    %    @    &
```
If you need to use those characters, describe them within the script that sets to Cmdline resources.
"CommandLines" will appear. If you need to stop the user programs, select "StopCommands."
StopCommand is executed during Offline processing to stop user applications.
You do not always have to set up the StopCommand.
[StopCommand exit codes]
StopCommand has the following exit codes:
0: Normal exit. The Offline processing is successfully done.
Other than 0: Abnormal exit. The Offline processing fails. When the script exits with the cord other than 0, the resource will enter Faulted.
If you do not use StopCommand, start from step 6.
Note
If "none" is set to StopCommands, regardless of the settings of Flags, LIEOFFLINE attribute is enabled and CLUSTEREXCLUSIVE is disabled. In this status, the Cmdline resource is started and monitored.
Select "FREECHOICE" and then enter the full path of StopCommand. If you need to specify arguments, delimit them with blanks.
"CommandLines" will appear. Select "CheckCommands."
CheckCommand is executed periodically to notify RMS of the state of the user applications.
[CheckCommand exit codes]
CheckCommand has the following exit codes:
0: Online.
Other than 0: Offline.
If you enable the ALLEXITCODES attribute, the script that is defined in CheckCommand provides more detailed state of the user applications. Change the attribute in step 8.
Select "FREECHOICE" and then enter the full path of the CheckCommand. If you need to specify arguments, delimit them with blanks.
Change the attribute.
Change the attribute to suit the purpose. To change the attribute, select "Flags[0]." For details on the attribute, see "Table 6.2 Attributes of the Cmdline resource" in "6.11 Notes When Setting Cmdline Resources."
Note
If you enable the "NULLDETECTOR" attribute, CheckCommand is not started from RMS. For hot-standby operation, enable the following two attributes;
- STANDBYCAPABLE
  RMS executes Standby processing of the resources on all the nodes where the userApplication is Offline.
- ALLEXITCODES
  Check script provides the detailed state of the resource with the exit code.
  For further details about the hot-standby operation settings, see "6.6 Setting Up Online/Offline Scripts."
Finally, select "SAVE+EXIT."

6.7.3.2 Setting Up Fsystem Resources

An Fsystem resource must be set up if you want to mount a file system when userApplication is started.

For notes when setting Fsystem resources, see "6.12 Notes When Setting Fsystem Resource."

You need to set up the Fsystem resource according to the flow below:

Note

The file system on the volume of LVM (Logical Volume Manager) cannot be controlled in Fsystem resource.

Defining mount points
Define mount points for file systems that are used as resources in the /etc/fstab.pcl file on all the nodes where userApplication is configured. Each line must begin with "#RMS#."
The description format of the first field to the sixth field in a line beginning with "#RMS#" is the same as that of the /etc/fstab file of RHEL. For details, refer to the official documentation of RHEL.
Example: /etc/fstab.pcl file
```
#RMS#/dev/sdd1        /mnt/swdsk1          ext3     defaults        0 0
#RMS#/dev/sdd2        /mnt/swdsk2          ext4    nodelalloc      0 0
#RMS#/dev/sdd3        /mnt/swdsk3          xfs     defaults       0 0
```
If you plan to use GDS volumes, you need to define the /etc/fstab.pcl file as follows.
Example: /etc/fstab.pcl file
```
#RMS#/dev/sfdsk/class0001/dsk/volume0001 /mnt/swdsk1 ext3  defaults    0 0
#RMS#/dev/sfdsk/class0001/dsk/volume0002 /mnt/swdsk2 ext4  nodelalloc  0 0
#RMS#/dev/sfdsk/class0001/dsk/volume0003 /mnt/swdsk3 xfs   defaults    0 0
```
Note
- If you have defined the same device or mount point in the /etc/fstab file, those definitions can be removed by making them into comment lines. If those definitions are remained, userApplications may fail to be started normally.
- Ext4 and xfs are used to make the allocation of the disk area more efficiently, and to improve the writing performance, by using their "Delayed Allocation" feature. As a result of the implementation of "Delayed Allocation", there is a possibility that a part of data is lost by OS panic or power supply interruption of servers, because the sojourn time on the memory of data that should be stored on the disk becomes longer.
  When a program has to guarantee writing immediately after writing in file system, the application which writes the file should issue the fsync() call. Refer to Storage Administration Guide of the Red Hat, Inc. for "Delayed allocation."
- For the directory paths that are specified as the mount points, specify any paths that do not include symbolic links.
Starting the GDS volume (only if necessary)
If a file system or a file to be maintained exists in a volume managed by GDS, start the GDS volume in any one for nodes configuring a cluster.
Example: When starting the volume volume0001 of the disk class class with a command
```
# /usr/sbin/sdxvolume -N -c class -v volume0001
```
Confirming file systems are created and can be mounted
It is necessary to create file systems on disk partitions that are used as resources. Refer to Storage Administration Guide of the Red Hat, Inc. for "Create file systems".
Example: Creating the ext3 file system
Create the file system.
```
# /sbin/mkfs -t ext3 /dev/sdd1
```
Check if the file systems can be mounted.
```
# /bin/mount -t ext3 /dev/sdd1 /mnt/swdsk1
# /bin/umount /mnt/swdsk1
```
Example: Creating the ext4 file system
Create the file system.
```
# /sbin/mkfs.ext4 /dev/sdd2
```
Check if the file systems can be mounted.
```
# /bin/mount -t ext4 /dev/sdd2 /mnt/swdsk2
# /bin/umount /mnt/swdsk2
```
Example: Creating the xfs file system
Create the file system.
```
# /sbin/mkfs.xfs /dev/sdd3
```
Check if the file systems can be mounted.
```
# /bin/mount -t xfs /dev/sdd3 /mnt/swdsk3
# /bin/umount /mnt/swdsk3
```
Tuning file systems
Set up the cluster environment in according to their file system types.
- Forcible file system check prevention (recommended for ext3 and ext4)
  If ext3 or ext4 is used for a file system, the file system might forcibly be checked during Online processing of a switching file system. It is part of the ext3 and ext4 specification that file systems are checked when a certain number of mounting has been executed since the last file system check, or a certain period of time has passed.
  When the file systems are forcibly checked along with node startup or failover, timeout occurs due to file system Online processing, and PRIMECLUSTER startup or failover might fail.
  It is necessary to prevent the file systems from being checked by executing the following command for all the ext3 and ext4 switching files.
  Example: Configuring and confirming the prevention of file systems from being checked
```
# /sbin/tune2fs -c0 -i0 /dev/sdd1
```
  After executing the above command, check if "Maximum mount count :-1", "Check interval:0"is displayed using the following command:
```
# /sbin/tune2fs -l /dev/sdd1 | /bin/grep "Maximum mount count"
Maximum mount count:      -1
# /sbin/tune2fs -l /dev/sdd1 | /bin/grep "Check interval"
Check interval:           0 (<none>)
```
  If the forcible file system check is prevented, file systems might corrupt due to failures such as disk errors and kernel bug. These failures cannot be detected through file system logging and journaling. The file system corruption might cause data corruption.
  To prevent this, execute the "fsck - f" command to enable the file system forcible check during periodic maintenance.
- Set Delayed Allocation disabled. (Only for ext4)
  For ext4 file systems, Delayed Allocation feature can be disabled by specifying nodelalloc for the mount attribute.
  You need to specify mount attribute field in the /etc/fstab.pcl file as follows.
```
#RMS#/dev/sdd2        /mnt/swdsk2          ext4    nodelalloc     0  0
```
Stopping the GDS volume (Only when Step 2 has already been implemented)
Stop the GDS volume started in Step 2.
Example: Stopping the volume volume0001 of the disk class class with a command
```
# /usr/sbin/sdxvolume -F -c class -v volume0001
```

Registering Fsystem resource to Cluster Applications

Select "LocalFileSystems" from "turnkey wizard STANDBY."

Settings of turnkey wizard "STANDBY" (APP1:not yet consistent)
 1) HELP                                 9) Procedure:BasicApplication(-)
 2) -                                   10) Symfoware(-)
 3) SAVE+EXIT                           11) Procedure:SystemState3(-)
 4) -                                   12) Procedure:SystemState2(-)
 5) ApplicationName=APP1                13) Gls:Global-Link-Services(-)
 6) Machines+Basics(app1)               14) IpAddresses(-)
 7) CommandLines(-)                     15) LocalFileSystems(-)
 8) Procedure:Application(-)            16) Gds:Global-Disk-Services(-)
Choose the setting to process: 15

Select "AdditionalMountPoint."

File systems (Lfs_APP1:not yet consistent)
1) HELP                   4) REMOVE+EXIT            7) (Timeout=180)
2) -                      5) AdditionalMountPoint
3) SAVE+EXIT              6) (Filter=)
Choose the setting to process: 5

The mount points, which are defined in the above-mentioned /etc/fstab.pcl file for preliminary setup, will appear. Select the mount point to be set in userApplication.

1) HELP                                 6) /mnt/swdsk2
2) RETURN                               7) /mnt/swdsk3
3) FREECHOICE
4) ALL
5) /mnt/swdsk1
Choose a mount point: 5

Select "SAVE+RETURN."

Set flags for mount point: /mnt/swdsk1 Currently set: LOCAL,AUTORECOVER (LA)
1) HELP                   4) DEFAULT                7) SHARE(S)
2) -                      5) SYNC(Y)                8) MONITORONLY(M)
3) SAVE+RETURN            6) NOT:AUTORECOVER(A)
Choose one of the flags: 3

If you register multiple mount points, repeat steps 2 to 4 for each mount point. After you have registered all necessary mount points, Select "SAVE+EXIT."

File systems (Lfs_APP1:consistent)
 1) HELP                                 6) MountPoints[0]=LA:/mnt/swdsk1
 2) -                                    7) MountPoints[1]=LA:/mnt/swdsk2
 3) SAVE+EXIT                            8) MountPoints[2]=LA:/mnt/swdsk3
 4) REMOVE+EXIT                          9) (Filter=)
 5) AdditionalMountPoint                10) (Timeout=180)
Choose the setting to process:  3

6.7.3.3 Preliminary Setup for Gds Resources

[Prerequisites]

If you need to set up a Gds resource, you must first set up a shared volume.
Also, before you make the settings required for the Gds resources, execute the following command on either node.

This command operation is required for linking the specified Gds resources with the status of the userApplication and to start and stop the GDS volume. In addition, this command sets all the nodes sharing the volume to Offline status.

# /opt/SMAW/SMAWRrms/bin/hvgdsetup -a [class-name]
 ...
Do you want to continue with these processes ? [yes/no] y

Information

To check the setup status of a shared volume, execute the following command:

# /opt/SMAW/SMAWRrms/bin/hvgdsetup -l

Note

If the preliminary setup is not performed, the cluster application is set to Inconsistent status. For details, see "Cluster applications become "Inconsistent"." in "Cluster System Related Error" of "PRIMECLUSTER Global Disk Services Configuration and Administration Guide."
This operation must not be performed when a GFS shared file system is used.

6.7.3.4 Setting Up Gds Resources

After completing the preliminary setup for the Gds resources, register the Gds resources to the userApplication. The procedure for setting Gds resources to a userApplication is described below.

Operation Procedure:

Select "Gds:Global-Disk-Services" from "turnkey wizard STANDBY."
Select "AdditionalDiskClass" from "Volume management."
If you want to register multiple resources, repeat steps 2 to 3 for each resource.
Example) When you register the class [CL] of the shared volume in userApplication:
A list of the cluster resources that can be registered in userApplication appears.
- In the case of the multiple-node cluster operation
  Select the cluster resource.
- In the case of the single-node cluster operation
  Enter the created class name of GDS by selecting "FREECHOICE", since the list of the cluster resources is not displayed.
Select "SAVE+EXIT."

6.7.3.5 Setting Up Gls Resources

[Prerequisites]

Before setting up the Gls resources, you must first set up the virtual interfaces and takeover virtual interfaces. For details, see "PRIMECLUSTER Global Link Services Configuration and Administration Guide: Redundant Line Control Function."

Operation Procedure:

Select "Gls:Global-Link-Services" from "turnkey wizard STANDBY."
Select "AdditionalTakeoverIpaddress" and then set the takeover IP address.
If you need to register multiple resources, repeat steps 2 to 4 for each resource.
The takeover IP addresses are displayed as options.
Select one.
Set up the operation mode of the resource.
The operation modes of the resources are "AUTORECOVER(A)" (selective value) and "NOT:AUTORECOVER(N)" (default value). If you select "SAVE+RETURN," the default value "NOT:AUTORECOVER(N)" will be set. Select "AUTORECOVER(A)" if an attempt to recover the resource is to be made for a given duration (default: 60 seconds) when a resource error occurs, or select "NOT:AUTORECOVER(N)" if switchover to another node is to be performed immediately.
To save the Gls resource settings and then exit, select "SAVE+EXIT."
You can change the timeout value of the Gls resource by selecting "(Timeout=60)" and setting any value (seconds).

See

By setting up the value in the StandbyTransition attribute when the cluster application is created, Gls resources on the standby node can be switched to the "Standby" state and the state of the Gls resources on the standby node can be monitored. For information on how to make this setting, see "6.7.2.1 Creating Standby Cluster Applications."

6.7.3.6 Setting Up Takeover Network Resources

[Prerequisites]

You need to configure the following information before using takeover network resources.

Define an IP address and a host name to be taken over between nodes
First, define a host address to be taken over between nodes. Next, allocate a host name to this IP address. The allocated host name should be defined in the /etc/hosts file for the node which uses the takeover IP address.
Define a target host
If you want to monitor the reachability of your network using ICMP while a takeover IP address is active, define a host name and an IP address of the device (e.g. router or layer 3 hub) which enables ICMP communication between more than one node in the /etc/hosts file. This process can be skipped if you do not need to monitor the reachability of the network.
Confirming a network interface
Confirm the network interface (e.g. eth2) on which the takeover IP address will be active is being recognized by the operating system and available.
Define a setting file for the cluster
Add the takeover IP address to the following file:
/usr/opt/reliant/etc/hvipalias
Enter entries according to the following rules.
```
<node name> <takeover> <interface> <netmask/prefix>
```
```
<node name> : CF node name of the node which uses the takeover IP address
<takeover>  : Host name of the takeover IP address
<interface> : Network interface name on which the takeover IP address will be activated
<netmask/prefix>   : Netmask for the takeover IP address (for IPv4), or network prefix length (for IPv6)
```
Example
When an IPv4 address for the host "takeover" (netmask 255.255.255.0) is taken over between two nodes (node0 and node1) on the network interface eth2, define as follows (specify the 8-digit netmask in hexadecimal).
```
node0 takeover eth2 0xffffff00
node1 takeover eth2 0xffffff00
```
When an IPv6 address for the host "takeover6" (network prefix length: 64) is taken over on the network interface eth3, define as follows.
```
node0 takeover6 eth3 64
node1 takeover6 eth3 64
```
Note
- An IPv6 link local address cannot be used as a takeover network resource. Moreover, it cannot be used as a communication destination of reachability monitoring.
- When defining a host name in the /etc/hosts file, do not assign the same host name to the IPv4 address and the IPv6 address.

Operation Procedure:

Select "IpAddresses" of turnkey wizard "STANDBY"

Settings of turnkey wizard "STANDBY" (APP1:not yet consistent)
1) HELP                                 9) Procedure:BasicApplication(-)
2) -                                   10) Symfoware(-)
3) SAVE+EXIT                           11) Procedure:SystemState3(-)
4) -                                   12) Procedure:SystemState2(-)
5) ApplicationName=APP1                13) Gls:Global-Link-Services(-)
6) Machines+Basics(app1)               14) IpAddresses(-)
7) CommandLines(-)                     15) LocalFileSystems(-)
8) Procedure:Application(-)            16) Gds:Global-Disk-Services(-)
Choose the setting to process:14

When you have previously specified the target host to monitor its network reachability using ICMP, select "AdditionalPingHost" and specify that target host.
```
Ipaddresses and ipaliases (Adr_APP1:not yet consistent)
1) HELP                   4) REMOVE+EXIT            7) (NeedAll=yes)
2) -                      5) AdditionalInterface    8) (Timeout=60)
3) SAVE+EXIT 6) AdditionalPingHost 9) (InterfaceFilter=)
Choose the setting to process:6
```
The target host name registered in the process of prerequisites will be shown as an option. Select the host name you have previously specified.
```
1) HELP
2) RETURN
3) FREECHOICE
4) router
5) l3hub
6) takeover
Choose another trusted host to ping:4
```
When you finish specifying the target host, you will be brought back to the previous screen. Since you are required to specify more than one target host, you need to select "AdditionalPingHost" again to add another target host on the previous screen.
Select "AdditionalInterface" to set up the takeover IP address.
When you have more than one IP address, you need to repeat this process for each IP address.
```
Ipaddresses and ipaliases (Adr_APP1:not yet consistent)
1) HELP                                 7) PingHostPool[0]=router
2) -                                    8) PingHostPool[1]=l3hub
3) SAVE+EXIT                            9) (NeedAll=yes)
4) REMOVE+EXIT                         10) (Timeout=60)
5) AdditionalInterface                 11) (InterfaceFilter=)
6) AdditionalPingHost
Choose the setting to process:
```
1. Takeover IP address registered in the process of prerequisites will be shown as an option.
  Select the host name for the takeover IP address you have previously specified.
```
1) HELP
2) RETURN
3) FREECHOICE
4) router
5) l3hub
6) takeover
Choose an interface name:6
```
2. Specify the operation mode for the takeover IP address.
```
Currently set: VIRTUAL,AUTORECOVER (VA)
1) HELP                   4) DEFAULT                7) MONITORONLY(M)
2) -                      5) BASE(B)                8) PING(P)
3) SAVE+RETURN            6) NOT:AUTORECOVER(A)
Choose one of the flags:
```
  - DEFAULT
    If you choose "DEFAULT", all values will revert back to their default values.
  - BASE, VIRTUAL
    This attribute is effective only when using an IPv4 address. When using an IPv6 address, do not change this attribute. The default value is "VIRTUAL".
    - BASE
    If you specify "BASE", activation/deactivation of the takeover IPv4 address and activation/deactivation of the physical interface (for example, eth2) are performed at the same time. "BASE" will be shown on "Currently set" and "5) VIRTUAL" is shown on the menu page.
    - VIRTUAL
    If you specify "VIRTUAL", activation/deactivation of the takeover IPv4 address and activation/deactivation of the logical interface (for example, eth2:1) are performed at the same time. "BASE" will be shown on the menu page. "VIRTUAL" will be shown on "Currently set" and "5) BASE" is shown on the menu page.
    You must activate the IPv4 address on the physical interface (for example, eth2) where the logical interface will be created beforehand because the takeover IPv4 address with this attribute specifies the IPv4 address for the logical interface. To activate the IPv4 address on the physical interface beforehand, make settings so that the IPv4 address is activated on the physical interface at startup of the operating system, or register the takeover IPv4 address with "BASE" attribute with the same takeover network resource.
  - AUTORECOVER, NOT:AUTORECOVER
    If you reactivate the takeover IP address, specify this attribute. The default value is "AUTORECOVER".
    - AUTORECOVER
    If you specify "AUTORECOVER" and the network interface goes down or becomes unreachable due to an error, it will try to activate the takeover IP address only once. "AUTORECOVER" will be shown on "Currently set"and "6) NOT:AUTORECOVER" is shown on the menu page. When the activation of the takeover IP address fails, it will be notified to the cluster.
    - NOT:AUTORECOVER
    If you specify "NOT:AUTORECOVER", the "AUTORECOVER" setting will be disabled. "NOT:AUTORECOVER" will be shown on "Currently set" and "AUTORECOVER" is shown on the menu page."
  - BASIC-IF
    You cannot use this attribute. Do not change.
  - MONITORONLY, NOT:MONITORONLY
    - MONITORONLY
    If you specify "MONITORONLY" and the network interface goes down or becomes unreachable due to an error, the error will not be notified to the cluster. "MONITORONLY" will be shown on "Currently set" and "7) NOT:MONITORONLY" is shown on the menu page. If you specify this attribute, a switchover due to a takeover IP address failure will not occur.
    - NOT:MONITORONLY
    If you specify "NOT:MONITORONLY", the "MONITORONLY" setting will be disabled. "NOT:MONITORONLY" will be shown on "Currently set" and "7) MONITORONLY" is shown on the menu page.
    Note
    At least one out of all the takeover IP addresses you have registered to the takeover network resources should be set to "NOT:MONITORONLY".
  - PING
    By setting this attribute, you can specify the previously configured target host for the takeover IP address. Select the target host name to be monitored which you have set in the process of prerequisites.
```
1) HELP
2) RETURN
3) router(000)
4) l3hub
Choese a ping host of the pool ():3
```
Change "Timeout" if needed.
Use "Timeout" to specify the timeout value (in seconds) for the system to wait until all registered takeover IP addresses become active or inactive. When any takeover IP address does not become active or inactive after the timeout value elapses, the resource will notify the error to the cluster. The default value is 60 seconds. Specify the value bigger than 45 seconds.

Note

NeedAll, InterfaceFilter
You cannot use these attributes. Do not change.

6.7.3.7 Setting Up Procedure Resources

The procedure resource setting is used to register the state transition procedure resource of the products supported by PRIMECLUSTER in userApplication.

[Prerequisites]

To create a procedure resource, you must first create a state transition procedure and register the procedure to the resource database.

For details, see "D.1 Registering a Procedure Resource."

Operation Procedure:

Select "Procedure:XXXXXXXXXX" from "turnkey wizard STANDBY."
Example of registering cluster resources of the BasicApplication class to a userApplication:
Select "AdditionalProcedureResource" from "Application detail Resource wizard."
To register multiple resources, execute steps 2 to 4 for each resource.
The list of cluster resources will appear. Select one.
Note
If a cluster resource does not appear on this screen, it indicates that the cluster resource has not been registered in the resource database. Confirm whether the cluster resource has been registered on each node of the userApplication, which is designed with "6.7.2 Setting Up userApplication." Register cluster resources if they are not registered. For details on the "clgettree" command, see the manual pages of this command. For details on registering the cluster resource in the resource database, see "D.1 Registering a Procedure Resource."
You can change the following on this screen. If necessary, select "SAVE+RETURN" from "Application detail Resource wizard" after that.
- Timeout value of the state transition procedure
  The default value is 1,800 seconds. If you use a machine that requires more than 1800 seconds for timeout, you need to change the value by selecting "TIMEOUT."
- Priority within the resource class
  The default value is specified by -p option when the state transition procedure resource is registered with the "claddprocrsc" command. If the -p option is not specified, 65535 is used as the default value. If you register multiple resources of the same class in the cluster application and specify the order of Online and Offline, change this value by selecting "PRIORITY" from the following screen. The resources will be Online in the ascending order, and will be Offline in the descending order.