Create the following files and scripts to register, control and monitor Systemwalker Operation Manager as an MC/ServiceGuard package:
Package configuration file
Package control script
Package monitoring script
Create the package configuration file on either the active node or the standby node.
Create package control scripts and package monitoring scripts on each active and standby node, placing them in the same location in each case (using the same directory and path). Do not create script files on the shared disk. With MC/ServiceGuard, it is recommended that scripts be created and stored under the "/etc/cmcluster" directory due to security reasons. After creating scripts, be sure to assign execution rights to them.
Samples of each of these files and scripts are provided.
Copy these samples, and then modify them to match the environment.
Create a package configuration file that defines the Systemwalker Operation Manager Jobscheduler and Job Execution Control daemons as MC/ServiceGuard application services.
The following sample package configuration file is provided for Systemwalker Operation Manager:
/opt/FHPJMCMN/etc/script/omgr.conf |
Copy the sample file, and then modify it to match the cluster system operation.
A package configuration file must be created for each active node, so create one file for 1:1 active/standby configurations, "N" files for N:1 active/standby configurations and two files for dual node mutual standby configurations.
The following section explains the definition information for the package configuration file and shows examples of customizing the definition information.
Definition information for the package configuration file
This section shows the information required to create a package configuration file, as well as the items that need to be modified to match the cluster system environment. The sample files presented here are based on MC/ServiceGuard A.11.09. If different version is used, refer to the MC/ServiceGuard manual and change the definition names that correspond to the version of the cluster where Systemwalker Operation Manager is used.
Definition name | Value in the sample file | How to modify item to match the environment |
|---|---|---|
PACKAGE_NAME | omgr | Specify an arbitrary package name. This name will be recognized by MC/ServiceGuard as the package name. |
FAILOVER_POLICY | CONFIGURED_NODE | - |
FAILBACK_POLICY | MANUAL | - |
NODE_NAME | hp01 | Specify the name of the node to be used as the active node. |
NODE_NAME | hp02 | Specify the name of the node to be used as the standby node. |
RUN_SCRIPT | /etc/cmcluster/omgr/omgr.sh | Specify the location of the package control script. |
RUN_SCRIPT_TIMEOUT | NO_TIMEOUT | - |
HALT_SCRIPT | /etc/cmcluster/omgr/omgr.sh | Specify the location of the package control script. |
HALT_SCRIPT_TIMEOUT | NO_TIMEOUT | - |
SERVICE_NAME | service1 | Specify an arbitrary service name. Use the same service name as the value of SERVICE_NAME[0]= in the package control script. |
SERVICE_FAIL_FAST_ENABLED | NO | - |
SERVICE_HALT_TIMEOUT | 300 | - |
SUBNET | xxx.xxx.xxx.xxx | Specify the subnet address. |
PKG_SWITCHING_ENABLED | YES | - |
NET_SWITCHING_ENABLED | YES | - |
NODE_FAIL_FAST_ENABLED | NO | - |
Example of how to make changes for N:1 active/standby configurations
For N:1 active/standby configurations, create "N" package configuration files, each with different names, and change the information in each file to match the environment.
In the following example, there are three active nodes.
Package configuration file 1: omgr1.conf
PACKAGE_NAME:omgr1 NODE_NAME:hp01 NODE_NAME:hpsub RUN_SCRIPT:/etc/cmcluster/omgr/omgr1.sh HALT_SCRIPT:/etc/cmcluster/omgr/omgr1.sh SERVICE_NAME:service1
Package configuration file 2: omgr2.conf
PACKAGE_NAME:omgr2 NODE_NAME:hp02 NODE_NAME:hpsub RUN_SCRIPT:/etc/cmcluster/omgr/omgr2.sh HALT_SCRIPT:/etc/cmcluster/omgr/omgr2.sh SERVICE_NAME:service2
Package configuration file 3: omgr3.conf
PACKAGE_NAME:omgr3 NODE_NAME:hp03 NODE_NAME:hpsub RUN_SCRIPT:/etc/cmcluster/omgr/omgr3.sh HALT_SCRIPT:/etc/cmcluster/omgr/omgr3.sh SERVICE_NAME:service3
Example of how to make changes for dual node mutual standby configurations
For dual node mutual standby configurations, create two package configuration files, each with different names, and change the information in each file to match the environment.
The following example shows how to change these files:
Package configuration file 1: omgr1.conf
PACKAGE_NAME:omgr1 NODE_NAME:hp01 NODE_NAME:hp02 RUN_SCRIPT:/etc/cmcluster/omgr/omgr1.sh HALT_SCRIPT:/etc/cmcluster/omgr/omgr1.sh SERVICE_NAME:service1
Package configuration file 2: omgr2.conf
PACKAGE_NAME:omgr2 NODE_NAME:hp02 NODE_NAME:hp01 RUN_SCRIPT:/etc/cmcluster/omgr/omgr2.sh HALT_SCRIPT:/etc/cmcluster/omgr/omgr2.sh SERVICE_NAME:service2
Package control scripts define information required to execute, monitor and respond to faults with application services registered with MC/ServiceGuard.
The following sample package control script is provided for Systemwalker Operation Manager:
/opt/FHPJMCMN/etc/script/omgr.sh |
Copy the sample file, and then modify it to match the cluster system operation.
The sample package control script is for 1:1 active/standby configurations (without subsystems). Changes need to be made for 1:1 active/standby configurations (with subsystems), the 1:1 active/standby configuration (with subsystems and partial cluster operation), and N:1 active/standby configurations and dual node mutual standby configurations.
After package control scripts with modified user-defined functions are created (one script for 1:1 active/standby configurations, "N" scripts for N:1 active/standby configurations and two scripts for dual node mutual standby configurations), the definition information in the package control scripts must be changed to match the actual operation.
The following section presents examples of how to change user-defined functions and the definition information in the package control script.
Changing user-defined functions
This section presents examples of how to change user-defined functions for 1:1 active/standby configurations (with subsystems), 1:1 active/standby configuration (with subsystems and partial cluster operation), and N:1 active/standby configurations and dual node mutual standby configurations.
Example of how to make changes for 1:1 active/standby configurations (with subsystems) and 1:1 active/standby configurations (with subsystems and partial cluster operation)
In a system where multiple subsystems are running, start/stop the daemons of Jobscheduler and Job Execution Control on each subsystem.
The following examples show a 1:1 active/standby configuration (with subsystems), in which both subsystem 0 and subsystem 1 are running, and a 1:1 active/standby configuration (with subsystems and partial cluster operation), in which subsystem 0 is not operated in a cluster but subsystem 1 is.
Change the "SUBSYSTEM" variable in the package control script to "PLU_SUBSYSTEM" and its value to the subsystem numbers. In the case of the 1:1 active/standby configuration (with subsystems and partial cluster operation), simply change the number for the subsystem that will be operated in the cluster.
[Before]
SUBSYSTEM="0"
[After]
1:1 active/standby configuration (with subsystems)
PLU_SUBSYSTEM="0 1"
1:1 active/standby configuration (with subsystems and partial cluster operation)
PLU_SUBSYSTEM="1"
Add the for, do, and done statements to the end of customer_defined_run_cmds to start Jobscheduler and Job Execution Control on each subsystem.
[Before]
# - 1:1 standby, N:1 standby, 2 nodes mutual standby sh /opt/FHPMJS/usr/lib/mjes/S99MJS -sys $SUBSYSTEM sh /opt/FHPJOBSCH/bin/S99JOBSCH -sys $SUBSYSTEM
[After]
# - 1:1 standby, N:1 standby, 2 nodes mutual standby for SUBSYSTEM in $PLU_SUBSYSTEM do sh /opt/FHPMJS/usr/lib/mjes/S99MJS -sys $SUBSYSTEM sh /opt/FHPJOBSCH/bin/S99JOBSCH -sys $SUBSYSTEM done
Add the for and do statements to the top of customer_defined_halt_cmds to stop Jobscheduler and Job Execution Control on each subsystem.
[Before]
function customer_defined_halt_cmds
{
# ADD customer defined halt commands.
: # do nothing instruction, because a function must contain some command.
MJSDAEMON=`ps -ef | grep "/usr/lib/mjes/mjsdaemon -sys $SUBSYSTEM" | grep -v "grep" | wc -l `[After]
function customer_defined_halt_cmds
{
# ADD customer defined halt commands.
: # do nothing instruction, because a function must contain some command.
for SUBSYSTEM in $PLU_SUBSYSTEM
do
MJSDAEMON=`ps -ef | grep "/usr/lib/mjes/mjsdaemon -sys $SUBSYSTEM" | grep -v "grep" | wc -l `Add the done statement to the end of customer_defined_halt_cmds.
[Before]
fi
cnt=`expr $cnt + 1`
sleep 3
done[After]
fi
cnt=`expr $cnt + 1`
sleep 3
done
doneIn the case of the 1:1 active/standby configuration (with subsystems and partial cluster operation), add the for, do, and done statements to the end of customer_defined_run_cmds so that the security information will be reflected on each subsystem automatically.
[Before]
# - 1:1 standby, N:1 standby ..../opt/FJSVfwseo/bin/mpaclcls ....sh /opt/FJSVfwseo/bin/jmacltrn.sh
[After]
# - 1:1 standby, N:1 standby ....for SUBSYSTEM in $PLU_SUBSYSTEM ....do ..../opt/FJSVfwseo/bin/mpaclcls -s $SUBSYSTEM ....sh /opt/FJSVfwseo/bin/jmacltrn.sh $SUBSYSTEM ....done
Example of how to make changes for N:1 active/standby configurations
For N:1 active/standby configurations, change the entries so that symbolic links are set up and canceled by the package control script.
Remove the comments (delete the hash "#") from the "# Make symbolic links. (if N:1 standby)" and "# remove symbolic links. (if N:1 standby)" sections.
[Before]
# Make symbolic links.(if N:1 standby)
# ACL Manager
#if [ ! "(" -h "/var/opt/FJSVfwseo/JM" -o -f "/var/opt/FJSVfwseo/JM" ")" ]
#then
# ln -s /disk1/FJSVfwseo/JM /var/opt/FJSVfwseo/JM
#fi
# Job Scheduler
#if [ ! "(" -h "/opt/FHPJOBSCH/db" -o -f "/opt/FHPJOBSCH/db" ")" ]
#then
# ln -s /disk1/FHPJOBSCH /opt/FHPJOBSCH/db
#fi
# Job Execution Control
#if [ ! "(" -h "/var/spool/mjes" -o -f "/var/spool/mjes" ")" ]
#then
# ln -s /disk1/FHPMJS/var/spool/mjes /var/spool/mjes
#fi
#if [ ! "(" -h "/etc/mjes" -o -f "/etc/mjes" ")" ]
#then
# ln -s /disk1/FHPMJS/etc/mjes /etc/mjes
#fi
# Calendar
#if [ ! "(" -h "/opt/FHPjmcal/post" -o -f "/opt/FHPjmcal/post" ")" ]
#then
# ln -s /disk1/FHPjmcal/post /opt/FHPjmcal/post
#fi
# Stem
#if [ ! "(" -h "/var/opt/FJSVstem" -o -f "/var/opt/FJSVstem" ")" ]
#then
# ln -s /disk1/FJSVstem /var/opt/FJSVstem
#fi
(omitted)
# remove symbolic links.(if N:1 standby)
# Job Scheduler
#if [ -h "/opt/FHPJOBSCH/db" ]
#then
# rm /opt/FHPJOBSCH/db
#fi
# Job Execution Control
#if [ -h "/var/spool/mjes" ]
#then
# rm /var/spool/mjes
#fi
#if [ -h "/etc/mjes" ]
#then
# rm /etc/mjes
#fi
# ACL Manager
#/opt/FJSVfwseo/bin/mpaclcls -u
#if [ -h "/var/opt/FJSVfwseo/JM" ]
#then
# rm /var/opt/FJSVfwseo/JM
#fi
# Calendar
#if [ -h "/opt/FHPjmcal/post" ]
#then
# rm /opt/FHPjmcal/post
#fi
# Stem
#if [ -h "/var/opt/FJSVstem" ]
#then
# rm /var/opt/FJSVstem
#fi[After]
# Make symbolic links.(if N:1 standby)
# ACL Manager
if [ ! "(" -h "/var/opt/FJSVfwseo/JM" -o -f "/var/opt/FJSVfwseo/JM" ")" ]
then
ln -s /disk1/FJSVfwseo/JM /var/opt/FJSVfwseo/JM
fi
# Job Scheduler
if [ ! "(" -h "/opt/FHPJOBSCH/db" -o -f "/opt/FHPJOBSCH/db" ")" ]
then
ln -s /disk1/FHPJOBSCH /opt/FHPJOBSCH/db
fi
# Job Execution Control
if [ ! "(" -h "/var/spool/mjes" -o -f "/var/spool/mjes" ")" ]
then
ln -s /disk1/FHPMJS/var/spool/mjes /var/spool/mjes
fi
if [ ! "(" -h "/etc/mjes" -o -f "/etc/mjes" ")" ]
then
ln -s /disk1/FHPMJS/etc/mjes /etc/mjes
fi
# Calendar
if [ ! "(" -h "/opt/FHPjmcal/post" -o -f "/opt/FHPjmcal/post" ")" ]
then
ln -s /disk1/FHPjmcal/post /opt/FHPjmcal/post
fi
# Stem (*1)
if [ ! "(" -h "/var/opt/FJSVstem" -o -f "/var/opt/FJSVstem" ")" ]
then
ln -s /disk1/FJSVstem /var/opt/FJSVstem
fi
(omitted)
# remove symbolic links.(if N:1 standby)
# Job Scheduler
if [ -h "/opt/FHPJOBSCH/db" ]
then
rm /opt/FHPJOBSCH/db
fi
# Job Execution Control
if [ -h "/var/spool/mjes" ]
then
rm /var/spool/mjes
fi
if [ -h "/etc/mjes" ]
then
rm /etc/mjes
fi
# ACL Manager
/opt/FJSVfwseo/bin/mpaclcls -u
if [ -h "/var/opt/FJSVfwseo/JM" ]
then
rm /var/opt/FJSVfwseo/JM
fi
# Calendar
if [ -h "/opt/FHPjmcal/post" ]
then
rm /opt/FHPjmcal/post
fi
# Stem (*1)
if [ -h "/var/opt/FJSVstem" ]
then
rm /var/opt/FJSVstem
fiChange "/disk1" to match the actual mount point of the shared disk.
Remove the comment only if the Master Schedule Management function is enabled.
Example of how to make changes for dual node mutual standby configurations
Change the value of the "SUBSYSTEM" variable in the package control script to "1".
[Before]
SUBSYSTEM="0"
[After]
SUBSYSTEM="1"
Change the "- 1:1 standby, N:1 standby" and "- 2 nodes mutual standby" section for dual node mutual standby. (Change the position of the hash "#".)
[Before]
# - 1:1 standby, N:1 standby /opt/FJSVfwseo/bin/mpaclcls sh /opt/FJSVfwseo/bin/jmacltrn.sh # - 2 nodes mutual standby #/opt/FJSVfwseo/bin/mpaclcls -s $SUBSYSTEM #sh /opt/FJSVfwseo/bin/jmacltrn.sh $SUBSYSTEM
[After]
# - 1:1 standby, N:1 standby #/opt/FJSVfwseo/bin/mpaclcls #sh /opt/FJSVfwseo/bin/jmacltrn.sh # - 2 nodes mutual standby /opt/FJSVfwseo/bin/mpaclcls -s $SUBSYSTEM sh /opt/FJSVfwseo/bin/jmacltrn.sh $SUBSYSTEM
Prepare another package control script with the same (changed) content but a different file name, and change the value of the "SUBSYSTEM" variable to "2".
[Before]
SUBSYSTEM="1"
[After]
SUBSYSTEM="2"
Definition information for package control scripts
This section shows the information required to create a package control script, as well as the items to be changed to match the cluster system environment. The sample files presented here are based on MC/ServiceGuard A.11.09. If different versions are being used, refer to the MC/ServiceGuard manual and change the definition names that correspond to the version of the cluster where Systemwalker Operation Manager is being used.
Definition name | Value in the sample file | How to modify item to match the environment |
|---|---|---|
VGCHANGE | "vgchange -a e" | - |
VG[0] | "vg02" | Specify an arbitrary volume group name. |
LV[0] | "/dev/vg02/lvol1" | Specify the shared disk device. |
FS[0] | "/disk1" | Specify the shared disk mount point. |
FS_MOUNT_OPT[0] | "" | - |
FS_UMOUNT_COUNT | 1 | - |
FS_MOUNT_RETRY_COUNT | 0 | - |
IP[0] | "yyy.yyy.yyy.yyy" | Specify the logical IP address. |
SUBNET[0] | "xxx.xxx.xxx.xxx" | Specify the subnet address. |
SERVICE_NAME[0] | "service1" | Specify an arbitrary service name. Use the same service name as the value of SERVICE_NAME in the package configuration file. |
SERVICE_CMD[0] | "/etc/cmcluster/omgr/omgr_fmonitor" | Specify the location of the package monitoring script. |
SERVICE_RESTART[0] | "-r 0" | - |
Example of how to make changes for N:1 active/standby configurations
For N:1 active/standby configurations, create "N" package control scripts, each with different file names, and change the information in each file to match the environment.
The following example shows the changes to be made if there are three active nodes.
Package control script 1: omgr1.sh
VG[0]="vg01" LV[0]="/dev/vg01/lvol1" FS[0]="/disk1" IP[0]="111.111.111.111" SERVICE_NAME[0]="service1" SERVICE_CMD[0]= "/etc/cmcluster/omgr/omgr_fmonitor1"
Package control script 2: omgr2.sh
VG[0]="vg02" LV[0]="/dev/vg02/lvol1" FS[0]="/disk1" IP[0]="111.111.111.112" SERVICE_NAME[0]="service2" SERVICE_CMD[0]= "/etc/cmcluster/omgr/omgr_fmonitor2"
Package control script 3: omgr3.sh
VG[0]="vg03" LV[0]="/dev/vg03/lvol1" FS[0]="/disk1" IP[0]="111.111.111.113" SERVICE_NAME[0]="service3" SERVICE_CMD[0]= "/etc/cmcluster/omgr/omgr_fmonitor3"
Example of how to make changes for dual node mutual standby configurations
For dual node mutual standby configurations, create two package control scripts, each with different file names, and change each file to match the environment.
The following example shows how to change these files:
Package control script 1: omgr1.sh
VG[0]="vg01" LV[0]="/dev/vg01/lvol1" FS[0]="/disk1" IP[0]="111.111.111.111" SERVICE_NAME[0]="service1" SERVICE_CMD[0]= "/etc/cmcluster/omgr/omgr_fmonitor1"
Package control script 2: omgr2.sh
VG[0]="vg02" LV[0]="/dev/vg02/lvol1" FS[0]="/disk2" IP[0]="111.111.111.112" SERVICE_NAME[0]="service2" SERVICE_CMD[0]= "/etc/cmcluster/omgr/omgr_fmonitor2"
By creating package monitoring scripts and specifying the package monitoring script file in the package control script, the Systemwalker Operation Manager daemon running on the cluster system can be monitored and failover triggers can be notified to the cluster system.
The following sample package monitoring script is provided for Systemwalker Operation Manager:
/opt/FHPJMCMN/etc/script/omgr_fmonitor |
Copy the sample file, and then modify it to match the cluster system operation.
The sample script has been coded to monitor the execution status of the Jobscheduler and Job Execution Control daemon processes, and to trigger a failover if both daemon processes have stopped. For operations that only monitor one daemon process or the other, modify the sample script to match the operation.
The sample package monitoring script is for 1:1 active/standby configurations (without subsystems) and N:1 active/standby configurations. Changes need to be made for the 1:1 active/standby configuration (with subsystems), the 1:1 active/standby configuration (with subsystems and partial cluster operation), and dual node mutual standby configurations. A package monitoring script must be created for each active node, so create one file for 1:1 active/standby configurations, "N" files for N:1 active/standby configurations, and two files for dual node mutual standby configurations.
The following section presents examples of how to modify the package monitoring script.
Example of how to make changes for 1:1 active/standby configurations (with subsystems) and 1:1 active/standby configurations (with subsystems and partial cluster operation)
In a system where multiple subsystems are running, monitor daemons on each subsystem. The following examples show a 1:1 active/standby configuration (with subsystems), in which both subsystem 0 and subsystem 1 are running, and a 1:1 active/standby configuration (with subsystems and partial cluster operation), in which subsystem 0 is not operated in a cluster but subsystem 1 is.
Change the "SUBSYSTEM" variable in the package monitoring script to "PLU_SUBSYSTEM" and its value to the subsystem numbers. In the case of the 1:1 active/standby configuration (with subsystems and partial cluster operation), simply change the number for the subsystem that will be operated in the cluster.
[Before]
SUBSYSTEM="0"
[After]
1:1 active/standby configuration (with subsystems)
PLU_SUBSYSTEM="0 1"
1:1 active/standby configuration (with subsystems and partial cluster operation)
PLU_SUBSYSTEM="1"
Add the for and do statements immediately after while do to monitor daemons on each subsystem.
[Before]
while /bin/true
do
# - 1:1 standby, N:1 standby, 2 nodes mutual standby
MJSDAEMON=`ps -ef | grep "/usr/lib/mjes/mjsdaemon -sys
$SUBSYSTEM" | grep -v "grep" | wc -l `[After]
while /bin/true
do
for SUBSYSTEM in $PLU_SUBSYSTEM
do
# - 1:1 standby, N:1 standby, 2 nodes mutual standby
MJSDAEMON=`ps -ef | grep "/usr/lib/mjes/mjsdaemon -sys $SUBSYSTEM" | grep -v "grep" | wc -l `Add the done statement before sleep 10.
[Before]
sleep 10 done
[After]
done
sleep 10
doneExample of how to make changes for N:1 active/standby configurations
For N:1 active/standby configurations, create "N" package monitoring scripts, each with different file names. There is no need to change the content of the script files.
The following example shows how to create the files if there are three active nodes.
Package monitoring script 1: omgr_fmonitor1
Package monitoring script 2: omgr_fmonitor2
Package monitoring script 3: omgr_fmonitor3
Example of how to make changes for dual node mutual standby configurations
For dual node mutual standby configurations, the script must be edited and two script files must be created.
Change the value of the "SUBSYSTEM" variable in the package monitoring script to "1".
[Before]
SUBSYSTEM="0"
[After]
SUBSYSTEM="1"
Prepare another package control script with the same (changed) content but a different file name, and change the value of the "SUBSYSTEM" variable to "2".
[Before]
SUBSYSTEM="1"
[After]
SUBSYSTEM="2"