8.2.1 Procedure for Node Expansion

Take the following steps to prepare the existing nodes.

Procedure

Preparing for unexpected failures, you need to back up the entire system of all existing cluster nodes, the PRIMECLUSTER system and the configuration files of each component.
1. Back up the entire system.
  1. Stop RMS by executing the following command on any one of the existing cluster nodes.
```
node1# hvshut -a
```
  2. Reboot all the existing cluster nodes from a single user mode.
```
node1# /usr/sbin/shutdown -g0 -i0 -y
  ....  
ok boot -s
  ....  

Type control-d to proceed with normal startup,
 (or give root password for system maintenance):
   ....
```
  3. Mount the file system on all the existing cluster nodes.
```
node1# mountall -l
node1# zfs mount -a
```
  4. Back up the entire system or property in the shared disk by executing the "ufsdump(1M)" or "dd(1M)" command.
2. Back up the PRIMECLUSTER system and the configuration files of each component.
  1. Back up the configuration files of the PRIMECLUSTER system on all existing cluster nodes. See "Chapter 13 Backing Up and Restoring a PRIMECLUSTER System".
  2. Back up the configuration files that are used for GLS on all existing cluster nodes.
    To back up the configuration files for GLS (redundant line control), use the following command (For details on the "hanetbackup" command, see "PRIMECLUSTER Global Link Services Configuration and Administration Guide: Redundant Line Control Function"):
```
node1# /opt/FJSVhanet/usr/sbin/hanetbackup -d /var/tmp/backup 
```
    To back up the configuration files for the GLS multipath function:
```
node1# cd /etc/opt/FJSVmpnet
node1# tar cvf - conf | compress > /var/tmp/backup/mpnetfile.tar.Z
```
Define the following information in the /etc/inet/hosts file of all the existing cluster nodes.
- The IP address of the node to be added and the name of the host that is associated with the IP address
- The IP address of the remote console that is connected to the node to be added and the host name that is associated with the IP address
Edit the "/opt/SMAW/SMAWRrms/bin/hvenv.local" file as shown below so that RMS does not start automatically on any of the existing cluster nodes, even when a cluster node is rebooted.
```
node1# vi /opt/SMAW/SMAWRrms/bin/hvenv.local
```
```
export HV_RCSTART=0
```
To add a new node, all the existing cluster nodes must be turned off.

8.2.1.3 Connecting a Node

Join a new node with the existing cluster nodes.

Procedure

Connect the LAN, Cluster Interconnect, and the RCI of the new node to the existing cluster environment.
At this time, configure the RCI address for the new node.
(This operation is done by your Fujitsu CE.)
After setting the RCI address, boot up the existing cluster nodes and check that no error message is output to the console or syslog.
Boot the new node and confirm that the new node and its remote console are correctly configured in the network by executing the "ping(1M)" command.
Confirm that the RMS is stopped on any one of the existing cluster nodes, and then stop the SF by executing the following commands on each existing cluster node.
Confirm that RMS is stopped.
```
node1# hvdisp -a
hvdisp: RMS is not running
```
Stop SF. Execute the following command on all the existing cluster nodes.
```
node1# sdtool -e
node1# sdtool -s
(SMAWsf, 30, 13) : The RCSD is not running
```
If the GFS shared file system is used in an existing node, take the following steps to stop the GFS operation.
1. Execute the following command for the entire GFS shared file system on any one of the existing cluster nodes, and then unmount the file system.
```
node1# sfcumntgl <mount point>
```
2. Execute the following command, and then stop the GFS daemon on all cluster nodes.
```
node1# sfcfrmstop
```

8.2.1.4 Configure Web-Based Admin View

This section explains how to configure Web-Based Admin View.

The nodes on which you need to configure Web-Based Admin View vary depending on the following cases;

Target node:

When the existing management server is used
The management server must be defined on the new node.
Configure Web-Based Admin View on the new node.
When the new node is used as the management server
The definition of the new management server must be defined on all the nodes.
Configure Web-Based Admin View on all the nodes.

Procedure

Set up Web-Based Admin View on the node.
See "4.2.3.1 Initial setup of the operation management server."
Confirm that Web-Based Admin View is running correctly.
For confirmation, use any one of the cluster nodes as explained in "4.2.3.2 Confirming Web-Based Admin View Startup."
Change the connection target information of the Java application.
Refer to "3.1.3.2 Setting up Java application" in "PRIMECLUSTER Web-Based Admin View Operation Guide" and change the IP addresses of the connection target management server retained in the shortcut and the desktop shortcut of the Java application.

8.2.1.5 Setting Up CF and SF

Make the CF and SF configuration by using Cluster Admin. This section explains how to configure CF and SF. See "1.1.4 Example of creating a cluster" in "PRIMECLUSTER Cluster Foundation (CF) Configuration and Administration Guide."

Procedure

Connect to the primary management server to display Web-Based Admin View.
Log on to Web-Based Admin View and then select the "Global Cluster Services" menu (see Figure 3).
Select a node name to be added on the node selection screen (see Figure 6).
Click the "Configure" button in the left-hand side panel on the screen, to start the CF wizard (see Figure 8).
Select an existing cluster system name from the "Create/Join Cluster" screen and then click the "Add local node to an existing CF Cluster" option button. Then, choose the "Next" button (see Figure 10).
Select a node that you want to add from [Available Nodes] on the "Selecting cluster nodes and the cluster name" screen, and then add the node to [Clustered Nodes] (see Figure 11).
When the Cluster Interconnect confirmation screen appears, confirm that the combination of network interface cards is correct on all nodes. Then, click the "Next" button (see Figure 14).
Check the "For RMS" checkbox for the CIP subnet settings (note that the RMS cannot use the CIP if this is not set.)
When the "Complete Configuration" dialog box appears, close that screen and click the "Finish" button. This completes the CF settings.
Configure the shutdown agent of the new node.
In SPARC M10 and M12, XSCF is used. Or when SPARC Enterprise M3000, M4000, M5000, M8000, M9000, SPARC Enterprise T1000, T2000, T5120, T5220, T5140, T5240, T5440, SPARC T3, T4, T5, T7, S7 series are used, configure the shutdown agent with the command.
This procedure is not necessary with other environments.
1. In SPARC M10 and M12, XSCF shutdown agent is used.
  Execute the following command on the new node and configure the XSCF shutdown agent.
```
node3# /etc/opt/FJSVcluster/bin/clsnmpsetup -a xscf <PPAR-ID> <domain-name> <XSCF-name1> <XSCF-name2> <user-name>
Enter User's Password: <password>
Re-enter User's Password: <password>
```
  PPAR-ID
  Specify the identification ID of the physical partition (PPAR) in which the logical domain of the cluster node belongs to.
  Make sure to enter "0" for SPARC M10-1, M10-4, and M12-2.
  For SPARC M10-4S and M12-2S, specify the integer value ranged from 0 to 15.
  Domain-name
  Specify the logical domain of the cluster node.
  Domain name can be checked from the virtinfo -a command.
  XSCF-name1
  Specify the host name or IP address for XSCF-LAN#0 of the cabinet in which the logical domain of the cluster node exists.
  Available IP addresses are IPv4 addresses.
  For SPARC M10-4S and M12-2S environments, specify the XSCF takeover IP address.
  XSCF-name2
  Specify the host name or IP address for XSCF-LAN#1 of the cabinet in which the logical domain of the cluster node exists.
  Available IP addresses are IPv4 addresses.
  For SPARC M10-4S and M12-2S environments, specify the XSCF takeover IP address.
  User-Name
  Enter the user name to log in to the XSCF of the cabinet where the logical address of the cluster node exists.
  Password
  Enter the password to log in to the XSCF of the cabinet where the logical address of the cluster node exists.
2. In SPARC Enterprise M3000, M4000, M5000, M8000 and M9000, XSCF shutdown agent is used.
  Execute the following command on the new node and configure the XSCF shutdown agent.
```
node3# /etc/opt/FJSVcluster/bin/clrccusetup -a xscf <XSCF-name1> <XSCF-name2> <user-name>
Enter User's Password: <password>
Re-enter User's Password: <password>
```
  XSCF-name1
  Enter the IP address of XSCF-LAN#0 or the host name that is registered in the /etc/inet/hosts file.
  Available IP addresses are IPv4 addresses.
  XSCF-name2
  If the number of IP address of the existing node of XSCF is 1, no need to specify the XSCF-name2.
  Enter the IP address of XSCF-LAN#1 or the host name that is registered in the /etc/inet/hosts file.
  Available IP addresses are IPv4 addresses.
  User-Name
  Enter a user name to log in to XSCF.
  Password
  Enter a password to log in to XSCF.
3. In SPARC Enterprise T5120, T5220, T5140, T5240, T5440, SPARC T3, T4, T5, T7 and S7 series, ILOM shutdown agent is used.
  Execute the following command on the new node and configure the ILOM shutdown agent.
```
node3# /etc/opt/FJSVcluster/bin/clrccusetup -a ilom <ILOM-name> <user-name>
Enter User's Password: <password>
Re-enter User's Password: <password>
```
  ILOM-Name
  Enter the IP address of ILOM or the host name of ILOM that is registered in the /etc/inet/hosts file.
  Available IP addresses are IPv4 and IPv6 addresses.
  IPv6 link local addresses are not available.
  User-Name
  Enter a user name to log in to ILOM.
  Password
  Enter a password to log in to ILOM.
4. In SPARC Enterprise T1000 and T2000, ALOM shutdown agent is used.
  Execute sfcipher(1M) command on the existing node to encrypt the password of ALOM of the new node.
```
node1# /opt/SMAW/SMAWsf/bin/sfcipher -c
Enter User's Password: <- Enter password of ALOM of the new node
Re-enter User's Passwd: <- Re-enter password of ALOM of the new node
D0860AB04E1B8FA3 <- Encrypted password
```
  Next, add the following description to the /etc/opt/SMAW/SMAWsf/SA_sunF.cfg file of the new node.
```
system-controller-alom-2k <ALOM-name> <user-name> <password>
void void <CF-node-name>
```
  ALOM-Name
  Enter the IP address of ALOM of the new node.
  User-Name
  Enter a user name to log in to ALOM of the new node.
  Password
  Enter a password to log in to ALOM of the new node.
  Enter the password encrypted by sfcipher(1M) command.
  CF-node-name
  Enter the CF node name of the new node.
Configure SF.
Configure SF to use the shutdown facility wizard.
For details, see "5.1.2 Configuring the Shutdown Facility."
Note
When starting shutdown facility wizard or after selecting shutdown agent in the selected screen of shutdown agent, the following screen may be displayed when the "Next" button is clicked.
When this screen is displayed, select the existing cluster nodes and click the "Done" button.

Confirm that the CF and SF have been configured correctly by executing the following commands on the GUI screen or on any one of the cluster nodes.

node1# cftool -n
Node    Number State     Os      Cpu
node1   1      UP        Solaris Sparc
node2   2      UP        Solaris Sparc
node3   3      UP        Solaris Sparc

node1# sdtool -s
Cluster Host    Agent                SA State      Shut State  Test State  Init State
------------    -----                --------      ----------  ----------  ----------
node1           SA_xscfsnmpg0p.so    Idle          Unknown     TestWorked  InitWorked
node1           SA_xscfsnmpg1p.so    Idle          Unknown     TestWorked  InitWorked
node1           SA_xscfsnmpg0r.so    Idle          Unknown     TestWorked  InitWorked
node1           SA_xscfsnmpg1r.so    Idle          Unknown     TestWorked  InitWorked
node2           SA_xscfsnmpg0p.so    Idle          Unknown     TestWorked  InitWorked
node2           SA_xscfsnmpg1p.so    Idle          Unknown     TestWorked  InitWorked
node2           SA_xscfsnmpg0r.so    Idle          Unknown     TestWorked  InitWorked
node2           SA_xscfsnmpg1r.so    Idle          Unknown     TestWorked  InitWorked
node3           SA_xscfsnmpg0p.so    Idle          Unknown     TestWorked  InitWorked
node3           SA_xscfsnmpg1p.so    Idle          Unknown     TestWorked  InitWorked
node3           SA_xscfsnmpg0r.so    Idle          Unknown     TestWorked  InitWorked
node3           SA_xscfsnmpg1r.so    Idle          Unknown     TestWorked  InitWorked

8.2.1.6 Setting Up CRM (Resource Database)

This section explains how to set up the Customer Resource Management (CRM) resource database.

Procedure

Reconfigure the resource database on the existing nodes.
Confirm the following:
- All of the existing nodes have been started.
- CIP entry of the new node is in /etc/cip.cf of the existing nodes.
  The resource database uses /etc/cip.cf to associate CF node names with CIP names.
Reconfigure the resource database.
The procedure is shown below:
1. Log into any one of existing nodes using a system administrator access privilege.
2. Specify the "-a" and the "-g" options in the "clsetp(1M)" command.
```
# /etc/opt/FJSVcluster/bin/clsetup -a node -g file
```
  As a result of this operation, a new node will be added to the resource database of the existing nodes.
  The configuration information on the resource database, created after the execution of the "clsetup(1M)" command, is used when the resource database of the new node is configured. Therefore, do not specify a directory that will be automatically deleted when rebooting the node with the "-g" option (for example: /tmp).
  Specify the CF node name in node, and a full path name of the file name of the resource database configuration information. "tar.Z" extension will be appended to the resource database configuration information.
  For example, to add a new node which has a CF node name of fuji4, and a configuration information file name of /mydir/rdb, the command to be executed will be as shown below:
```
# /etc/opt/FJSVcluster/bin/clsetup -a fuji4 -g /mydir/rdb
```
  The configuration information file of the resource database will be created as /mydir/rdb.tar.Z.
3. Confirm that the new node has been added to the resource database.
  Execute the "clgettree(1)" command, and then confirm that the new node is displayed on the output result. At this time, the state of the new node is displayed as UNKNOWN.
Set up the resource database of the new node.
Before setting up the resource database of the new node, confirm the following:
- The content of /etc/cip.cf of the new node must be the same as that of the existing nodes.
  Confirm that the CIP of the new node is in /etc/cip.cf of the new node and that the content of /etc/cip.cf is the same as that of the existing nodes.
- Communication must be enabled in CIP
  Confirm that the new node is connected to all of the existing nodes by CIP using the "ping(1M)" command.
  If two or more CIPs are configured in the new nodes, use the first CIP for the resource database. Then, connection will be enabled. An example using "fuji4RMS" as the new node is shown below:
```
# ping fuji4RMS
```
  After confirming the above, set up the resource database of the new node.
  The procedure is as follows:
1. Log into the new node using a system administrator access privilege.
2. Copy the "resource database configuration information" file created in Step 2) in "Recreate the settings for the resource database of the existing nodes" to the new node.
3. Specify the -s option in the clsetup(1M) command, and execute it.
```
# /etc/opt/FJSVcluster/bin/clsetup -s file
```
  Specify file with a full path name of the resource database configuration file.
  When the resource database configuration information file "rdb.tar.Z" is copied to /mydir, the command to be executed will be as shown below:
```
# /etc/opt/FJSVcluster/bin/clsetup -s /mydir/rdb.tar.Z
```
4. Confirm that the resource database of the new node is configured.
  Execute the clgettree(1) command for the new node and confirm the following:
  - The new node is displayed.
  - The state of the new node is displayed as ON.
  - The output result is the same as that of the existing nodes.
Register the hardware, which is connected to the new node, to the resource database.
Log into any one of nodes using a system administrator access privilege, and execute the command shown below:
```
# /etc/opt/FJSVcluster/bin/clautoconfig -r
```

Setting up synchronization with the resource database

If the individual nodes are restarted at different times after node expansion, the tuning parameter must be set up to be synchronized with the resource database. For details, see "3.5.1 Start up synchronization and the new node" in "PRIMECLUSTER Cluster Foundation (CF) Configuration and Administration Guide."

8.2.1.7 Setting Up GDS

If a new node was added to the GDS configuration, it is necessary to change the class scope by executing the sdxattr command.

For information on the change procedure, see "PRIMECLUSTER Global Disk Services Configuration and Administration Guide."

8.2.1.8 Setting Up RMS

This section explains how to register a new node (SysNode) in a userApplication that has been running on an existing node.

Procedure

Configuration for each resource
Take the following steps depending upon the resources in the existing userApplication:
- Cmdline
  Create the Start, Stop, and Check scripts in the new node or copy them from the existing node. If "Path enter" is selected from "Creation method" when creating the existing Cmdline resource, specify the paths to the scripts. If "New" is selected, the scripts under /opt/FJSVwvucw/scripts/start, /opt/FJSVwvucw/scripts/stop, and /opt/FJSVwvucw/scripts/check must be stored in the same directory of the new node. You also need to add the access privilege by executing the "chmod(1)" command.
- Gds
  Take the following steps to expand the cluster scope:
  1. Expand the class scope.
    See "PRIMECLUSTER Global Disk Services Configuration and Administration Guide."
  2. Execute the following command on any one of the cluster nodes:
```
# /opt/SMAW/SMAWRrms/bin/hvgdsetup -a class
hvgdsetup with -a option performs the following tasks on
nodes to which the specified disk class belongs.
  1) Make GDS disk class on resource database not activated
     automatically when the node boots. If this operation has
      been done before, nothing will be performed anymore.
  2) Next make volumes of the specified disk class
     enabled manual online on a node on which an application
     is offline or faulted while the application is online
     or standby on another node.
  3) Then make volumes of the specified disk class stopped
     immediately.
     This process is executed on the nodes to which the disk
     class belongs.
Do you want to continue with these processes ? [yes/no] yes
```
    The following message might appear after executing the "hvgdsetup" command. This does not disrupt ongoing operation.
```
FJSVcluster: error: clrmd: 7516: An error occurred in the resource deactivation processing. (resource:resource rid:rid detail:detail)
WARNING !!
Failed to control 'dc_class' in the following node(s).
  node(s) node_name:  
Check the state of the nodes. If any nodes have failed, you may ignore this message.
```
- Fsystem
  Add the mount point entry to /etc/vfstab.pcl on the new node.
- Gls
  Take the following steps for Gls:
  1. Set up the virtual interface for the takeover IP address on the new node and register it as a cluster resource. For details, see the "PRIMECLUSTER Global Link Services Configuration and Administration Guide: Redundant Line Control Function."
  2. Restart Gls by executing the following command:
```
node3# /opt/FJSVhanet/usr/sbin/resethanet -s
```
- Takeover network
  Nothing needs be done at this time. In Procedure 3, however, it is necessary to recreate the resources.
- Procedure
  Create a state transition procedure on the new node and register the procedure resource with the cluster resource manager. For more details, see "E.1 Registering a Procedure Resource".
- Process monitoring
  Add the startup command to the new node. Also, you need to add the access privilege by using the "chmod(1)" command. Then, recreate the resources in Procedure 3.
Deleting userApplication
Delete the existing userApplication by using the userApplication Configuration Wizard. At this time, select "Delete only userApplication."
For more details, see "10.3.1 Changing the Cluster Application Configuration."
Recreating the takeover network and process monitoring resources
If the takeover network resource and the process monitoring resource are registered in the cluster system, first delete and then recreate those resources.
See "10.5 Deleting a Resource," "Creating Takeover Network Resource," and "Creating Process Monitoring Resources."
Recreating userApplication
Recreate the userApplication that was deleted in Procedure 2, using the same procedure as that used to create it. Note that the new node must be registered when SysNode is to be selected. For details, see "Creating Cluster Applications."
Copy /opt/SMAW/SMAWRrms/bin/hvenv.local of the existing node to /opt/SMAW/SMAWRrms/bin/ of the new node.
Edit /opt/SMAW/SMAWRrms/bin/hvenv.local in each node with the "vi" editor, and delete the following entry:
```
export HV_RCSTART=0
```