The following table summarizes scalable controller states according to the states of its controlled applications:
Controller state | Child application states |
|---|---|
Online | At least one online on any machine |
Offline | All offline on all machines |
Faulted | At least one faulted on any machine, all others offline on all machines |
Standby | At least one standby on any machine, none online on any machine |
Warning | At least one online on any machine, others offline on all machines |
Note: Although two or more different scalable applications can be online at the same time on the same or on different hosts, each scalable application can still be online only on a single host. For example, when creating an RMS configuration for Oracle RAC, the Oracle daemon processes that must be online on multiple hosts should be represented not by a single RMS application, but by a number of separate, controlled RMS applications - one child application per daemon. The parent scalable controller for Oracle RAC becomes a single point of control, administration, and display for all these child applications.
Controlling applications and controlled applications may become the online state on different nodes. However, monitoring and controlling for applications can work normally and no action is needed.
Note
When you restart a node after it stopped abnormally because of a node failure, an application which is controlled from multiple nodes may become the online state. However, it has no problem operating the system and no action is needed.
Priority Online requests from the controller are propagated to all controlled applications, each of which will attempt to come Online according to the sequence of hosts in their PriorityList attribute. Offline and Standby requests are propagated to all controlled applications on all the hosts.
The attribute IndependentSwitch, which must be set to 1 for a scalable controller object, assures that a parent application can be switched from host to host without affecting controlled child applications. However, the IndependentSwitch setting is ignored when the parent application is switched via a forced request such as 'hvswitch -f'; in this case the controller will propagate Offline request to each child application as a part of its switchover processing.
When a parent application and child applications are switched over simultaneously by shutting down RMS or the operating system and restarting RMS is completed before the child applications status changed to Online on the secondary prior node, the child applications statuses will attempt to come Online according to the sequence of nodes in their PriorityList attributes. Therefore the child applications statuses will be Online again on the previous node that has not switched over.
Scalable controller displays Warning state in GUI and hvdisp command if one of the following conditions is met. This is to warn administrators that some scalable applications are not running.
When at least one application, which is controlled by an Online controller, was transited to an Offline, Standby, or Faulted state from an Online state
When at least one application, which is controlled by an Standby controller, was transited to an Offline or Faulted state from a Standby state
Note
When RMS has been stopped on all the nodes where some applications (among all applications controlled by an Online controller) can run, the controller will remain in the Online state because it cannot detect a failure.
An application that contains one or more scalable controllers will acquire a posted Warning state if at least one of its scalable controllers appears in the posted Warning state. It will revert to its respective posted state when all its scalable controllers go out of the posted Warning state. As was the case for the controller, the GUI and hvdisp will display this application posted Warning state to alert the administrator that the scalable parent application is not running in its full capacity.
The posted Warning state will be changed to a respective posted state when the controller transitions to offline of faulted, or when all controlled applications transition either to online or standby.
The controller's StateChangeScript attribute specifies a script to be executed for a scalable controller whenever a controlled child application transitions into an online, offline, faulted, or standby state. The script is executed on every host where the controlling parent application can run, even if it is done on behalf of a request originated from the controller itself. The state change script will also be executed if a host where a child application is running changes its state to offline or faulted. This script has no impact on state transitions - its abnormal exit code is merely recorded in the switchlog.
If more than one state transition event is delivered at the same time, then a state change script will be executed for each one of them. The state change script is scheduled immediately upon receiving an event, without any delay. For multiple events arriving from the same host, it is guaranteed that the order of state change script invocation will follow the order of the received events.
Script execution environment variables
The state change script's environment contains a set of variables that can be used for decision processing at runtime. Their names and purposes are summarized below. For a complete description of their content and format, see the "E.2 Global environment variables", the "E.4 Script execution environment variables".
RMS provides two complementary variables that are set according to whether the state change script was invoked due to an application state change or a host state change:
HV_APPLICATION_STATE_CHANGE_INFO
Set when a state change script is invoked because a controlled application changed its state. The colon-delimited substrings are:
Previous state of the application on its node, as recorded on the local node
Current state of the application on its node, as recorded on the local node
Name of the node where the application has changed its state
Name of the application that has changed its state
HV_APPLICATION_STATE_CHANGE_INFO will be empty if the state change script is invoked because a node changed its state.
HV_HOST_STATE_CHANGE_INFO
Set when a state change script is invoked for a scalable controller because a SysNode that can run a controlled child application changes its state to Offline or Faulted. The colon-delimited substrings are:
Previous node state
Current node state
Name of the node
Reason for the node state change: Shutdown Facility, hvshut command, or unknown.
HV_HOST_STATE_CHANGE_INFO will be empty if the state change script is invoked because a controlled child application changed its state.
Like other scripts invoked by RMS on behalf of an object, the state change script environment includes the following standard set of script variables:
HV_APPLICATION
Name of userApplication object at the top of the current sub-tree that contains the current object.
HV_AUTORECOVER
If set to 1, the script was initiated due to an AutoRecover attempt.
HV_FORCED_REQUEST
If set to 1, the script is currently processing a forced request.
HV_LAST_DET_REPORT
Last detector report for the current object.
HV_OFFLINE_REASON
Reason for ongoing offline processing: detect request, manual switchover, follow-up processing after a previous resource failure, or stopped application.
HV_NODENAME
Name of current object.
HV_SCRIPT_TYPE
Script type.
NODE_SCRIPTS_TIME_OUT
Timeout value for the current object and script type.
Script execution sequence
The state change script is executed immediately upon an event. As a result, it can run in parallel with other RMS scripts, including state change scripts for other objects or even for the current object. Since several events may be delivered nearly at the same time, the user's script is responsible to work in proper sequence. The writer of the script must assure proper access to shared resources from this and other scripts by using locks or other OS means.
The controller's ApplicationSequence attribute controls how online, offline, and standby requests are propagated to child applications. The attribute lists all the controller's child applications that are specified in the controller Resource attribute, but arranged in groups that are processed in parallel or sequentially:
A space-separated group of applications is called a request group. All the applications in a request group are processed in parallel.
Request groups separated by colons (:) are processed sequentially. Online or standby requests are processed left-to-right. Offline requests are processed right-to-left.
For example, suppose the ApplicationSequence attribute for a controller contains the following specification:
A1 A2:B:C1 C2
The controller would issue online requests as follow:
A1 and A2 (the first request group) would be processed in parallel - neither would wait for the other. However, both A1 and A2 must post their respective online state before the controller proceeds to the next group.
B (the second request group) would be processed next. It must post its online state before the controller proceeds to the next group.
C1 and C2 (the third request group) would finally be processed in parallel.
Offline requests would be processed in the reverse order; that is, C1 and C2 first, then B, and finally A1 and A2.
The order is only important during a request propagated to controlled applications from the scalable controller. Applications state changes due to any other reason, including manual or automatic switchover, disregard ApplicationSequence.
The order of offline processing is also maintained for local hvshut requests such as 'hvshut -l'. However, the order is disregarded for clusterwide requests such as 'hvshut -a': each host will take its applications offline independently.
Like other subapplications, scalable request groups are not processed during 'hvshut -L' and 'hvshut -A' operations.
Note that the propagation of requests from the scalable controller is also affected by the state of the hosts where the child applications may run. If no hosts are available for the applications from the same request group, then the request is not propagated. Instead, it is delayed until such a host becomes available, or until ScriptTimeOut expires.
A controlling application must be able to auto-startup even when some cluster hosts are offline. This is required to allow for some controlled child applications to come online on the partial cluster, in the order defined in their parent controller's ApplicationSequence attribute.
For example, if initially all cluster hosts are down, and some of them come up while others are in maintenance, controlled applications that are supposed to run only on the up sub-cluster must be brought online following a request from their scalable controller, regardless of the fact that other cluster hosts are currently offline or faulted. It is impossible to use 'hvswitch -f' in this case because a consultant may not be present, so the configuration must come up on its own.
The above is achieved by the attribute PartialCluster of the userApplication object. If set to 1, then the application can negotiate its online request within the currently online hosts, even if some other hosts, including the application's primary host, are offline or faulted. If set to 0, then application can negotiate its online request only when all hosts where it can possibly run are online (current behavior). The default value is 0.
For an application that contains a scalable controller (i.e. for a controlling application), PartialCluster can be set to 1 if the application graph has no cluster-exclusive resources; otherwise, PartialCluster must be set to 0. Each of its controlled applications must have its PartialCluster attribute set to 0, and its AutoStartUp attribute set to 0.
If controlling applications do not contain any objects other than the controller objects, having their PartialCluster attribute set to 1 is safe. Indeed, if two or more controlling applications auto-startup on different sub-clusters, they will not share any exclusive resources. As a part of their auto-startup, they will propagate online requests to their controlled applications that, in turn, may come online if they only require the hosts from the current sub-cluster.
However, if some controlled applications require hosts from several sub-clusters, then these applications will not come online, which is actually the desired effect. Note that their PartialCluster is 0, so the controlled applications that span subclusters refuse online processing.
Because scalable controllers delay propagation of the online request until hosts for the controlled applications from each application group come online, the order imposed by ApplicationSequence will be preserved even if two or more hosts simultaneously initiate online requests for the same scalable controller.
When a controlling application has its PartialCluster set, it can be switched online, automatically or manually (with or without the '-f' forced option) between the hosts of a sub-cluster. Again, this is safe for the controlling application itself because it has no real resources other than controllers. Also, this is safe for the controlled applications because their PartialCluster is set to 0 - they will not come online if they cross the subcluster boundary.
Attempting to manually take the last online child application offline with the 'hvutil -f' command would cause the parent controller to go into the faulted state. To prevent the user from creating a faulted scenario, RMS rejects taking the last controlled online application offline with the 'hvutil -f' command and generates the following message:
ERROR: Application controlled from another online application - <application>.
To illustrate controller behavior in maintenance mode, consider the following example:
The configuration consists of applications app1, app2, and app3 on nodes shasta1 and shasta2.
app2 controls app1 with a follow mode controller.
app3 controls app2 with a scalable mode controller.
The following figure shows the RMS configuration tree and the corresponding graph for this scenario.
Figure 2.6 Maintenance mode controller example
Switching applications between nodes
From the perspective of the graph, these may be thought of as "horizontal" switching operations.
If a scalable-mode child application is put into maintenance mode, the parent can still be switched between nodes with hvswitch, either manually or automatically as shown in the following figure. Note that putting app2 into maintenance mode automatically puts its follow-mode child application, app1, into maintenance mode as well.
Figure 2.7 Switching scalable parent with child in maintenance mode
Likewise, if a scalable-mode parent application is put into maintenance mode, the child can still be switched between nodes with hvswitch as shown in the following figure. However, in actual practice, this would only occur via manual commands because the parent issues no processing requests while in maintenance mode. Note that switching app2 to a different node automatically switches its follow-mode child application, app1, as well.
Figure 2.8 Switching scalable child with parent in maintenance mode
In summary, these "horizontal" hvswitch operations proceed normally.
Online, offline, and fault processing restrictions
From the perspective of the graph, these may be thought of as "vertical" switching operations.
Consider what happens if you attempt to take app3 offline with hvutil while app2 is in maintenance mode.
Figure 2.9 Attempting to take a parent offline with the child in maintenance mode
Unlike the hvswitch operations described previously, this 'hvutil -f' operation requires a response from its child application before it can proceed. Since maintenance mode prevents app2 from responding to normal processing requests from its parent, the offline request will eventually time out. The same situation occurs when fault processing via 'hvutil -c' is attempted.
Attempting to take a child offline while the parent is in maintenance mode as shown in the following figure also fails, but there is no long delay as in the previous case: RMS immediately generates an error because only the parent can initiate offline processing.
Figure 2.10 Attempting to take a child offline with the parent in maintenance mode
However, you can issue a fault processing 'hvutil -c' request to the child because that does not require a response from the parent.
In summary, these "vertical" hvutil operations will fail if the application requires a response from a child or parent application that is in maintenance mode.
Recommended approach for scalable controllers
It is highly recommended that maintenance mode requests be issued to the top-level application of a controlled hierarchy. This will help to avoid unexpected behavior:
Child applications will receive no automatic requests from the parent to switch to other nodes. Also, no automatic online, offline, or fault processing requests will propagate from the parent.
In case of a resource failure, a switch request from lower-level objects will fail immediately, so RMS can initiate fault processing without waiting for a timeout delay.
