5.4.3 Designing a Rule Definition

Define the filter rules to be used by the high-speed filter.

Considerations when creating filter rules

The items to consider when creating filter rules are as follows:

Unit of rule creation
Create a filter rule for each event type. Multiple filter rules cannot be defined simultaneously for a certain event type.
Processing pattern of filter rules
Select a suitable processing pattern from the ones described below, and then create a rule similar to the selected pattern.
Refer to Chapter 2, "Filter Rule Language Reference" in the Developer's Reference for information on filter rules.
Master data
Master data is referenced from within filter rules. Some individual events may contain only limited information such as an ID or code in order to cut down on the volume of event communication. Writing rules to process events is difficult in that situation, so it is possible to use master data that can be referenced using the ID or code as a key. Using master data in this way allows rules to be created more easily.
The master data must be created as files in CSV format by the user beforehand.
Refer to "5.4.4 Designing a Master Definition" for information on designing master data.
Memory usage for filter rules
Using filter rules requires a large amount of memory. Refer to "3.3.1.1 Amount of Memory when Using High-speed Filter Rules" and "3.3.1.2 Amount of Memory when Master Data is used by the High-speed Filter" for information on the memory required.

Processing pattern of filter rules

By defining filter rules in the high-speed filter, the user can describe event extraction as well as extraction and join processing in combination with master data. The output of the high-speed filter is used directly as the input of complex event processing.

The processes performed by the high-speed filter are generally represented by the following four patterns and their combinations:

5.4.3.1.1 Extraction process

This processing pattern extracts from the input events those events that meet the conditions described in IF-THEN statements in the filter rules.

Consider using this processing pattern if only the events required are to be extracted from massive volumes of events.

Example

Example of an extraction process

This is an example of extracting events using the content of the events (value).

"value > 10" is defined as the extraction condition.

Rule to be created

The rule to be created in the example above is as follows:

on inputEventTypeID {
    if ($value > 10) then output() as inputEventTypeID;
}

inputEventTypeID is the development asset ID of the target event type definition.

Information

Filtering the items in the events to be output

Items in the events to be output can also be filtered. Below is an example of outputting only "key" in the example shown above.

To output using a different format from that of the input events, a corresponding event type definition (filtered) will also be required.

Refer to Section 2.7, "Output Expression Format" in the Developer's Reference for details.

on inputEventTypeID {
    if ($value > 10) then output($key) as outputEventTypeID;
}

inputEventTypeID is the development asset ID of the target event type definition.
outputEventTypeID is the development asset ID of the event type definition (filtered) that represents the results of filtering the item.

5.4.3.1.2 Extraction process using master data matching

This processing pattern matches the relevant entries of master data (CSV files) on the basis of values such as those of the ID or code contained in the input events, and then extracts the events based on the values of the relevant entries.

Consider using this processing pattern if only the events required are to be extracted from massive volumes of events but the events themselves do not contain the information required for extraction.

To perform the processing of this pattern, a master definition must also be designed.

Example

Example of the extraction process using master data matching

This is an example of referencing the relevant entries of master data based on the "key" contained in the events, and then extracting the events based on the values of "address" in the entries. "address==Fukuoka" is defined as the extraction condition.

Rule to be created

The rule to be created in the example above is as follows:

on inputEventTypeID {
    if (lookup("masterDefinitionID", $key == $key, string($address)) == "Fukuoka") then output() as inputEventTypeID;
}

inputEventTypeID is the development asset ID of the target event type definition.
masterDefinitionID is the development asset ID of the master definition that corresponds to the master data to be referenced.
To compare "$address" as a string, use "string($address)" to fetch the values.
The left side of "$key == $key" is "key" of the input events and the right side is "key" of the master data.

Note

The master data information will not be assigned to the input events if they are only matched using "lookup". If join processing with master data is required, a join expression must be described. Refer to "5.4.3.1.3 Join processing with master data" below for information on join processing.

5.4.3.1.3 Join processing with master data

This processing pattern joins input events with master data. Consider using this processing pattern to assign the required data for the next complex event processing.

To pass the results of joining to complex event processing, an event type definition (filtered) corresponding to the join results will also be required.

Example

Example of join processing with master data

This is an example of joining the corresponding master data on the basis of "key" contained in the events, and then assigning "address" to the events.

Rule to be created

The rule to be created in the example above is as follows:

on inputEventTypeID {
    join("masterDefinitionID", $key == $key) output($key, "masterDefinitionID".$address) as outputEventTypeID;
}

inputEventTypeID is the development asset ID of the event type definition that is the rule target.
masterDefinitionID is the development asset ID of the master definition that corresponds to the master data to be referenced.
outputEventTypeID is the development asset ID of the event type definition (filtered) that represents the results of joining.

5.4.3.1.4 Weighting processing of text

This processing pattern can weight the text in input events by registering the weight of the keywords in the master data. This in turn allows applications including those that extract only those events with a total weighting that is above a threshold, and those that detect consecutively issued events that are above a threshold.

Example

Example of weighting processing of text

If the text contained in events contains search words that have been defined in the master data, assign the number of search words it contains as well as the total weighting value set for each search word, and then output them.

Rule to be created

The rule to be created in the example above is as follows:

on inputEventTypeID {
    join("masterDefinitionID", $message = $word)
        output($ID, 
               $subject,
               "masterDefinitionID".$word,
               "masterDefinitionID".$weight,
               lookup_count("masterDefinitionID".$word),
               lookup_sum("masterDefinitionID".$weight)) as outputEventTypeID;
}

inputEventTypeID is the development asset ID of the event type definition that is the rule target.
masterDefinitionID is the development asset ID of the master definition that corresponds to the master data to be referenced.
outputEventTypeID is the development asset ID of the event type definition (filtered) that represents the results of joining.

5.4.3.2 Complex Event Processing

Define the rules to be used by complex event processing.

Considerations when creating complex event processing rules

The items to consider when creating complex event processing rules are as follows:

Unit of rule creation
Create a complex event processing rule for each use application or for each purpose of performing event pattern detection. Decide on a unit of creation in which an event pattern detection described in a certain rule will not affect other rules.
It is also possible to describe multiple processes in one rule definition, but this will create large rule definitions and may lead to reduced maintainability.
For example, if events relating to home electronic equipment are to be processed and the content to be detected varies significantly between domestic appliances and information devices, create the following two rule definitions:
- Rule definition to detect patterns in events relating to domestic appliances
- Rule definition to detect patterns in events relating to information devices
Referencing external data
Consider whether referencing external data is necessary in event processing. An XTP cache can be used as external data.
Refer to "5.4.6 Designing XTP Collaboration" for information on referencing an XTP cache.
Whether processing results are to be sent or logged
The processing results of complex event processing rules can be sent to an external Web service using SOAP but they can also be logged in an event log using logging. Apply the SOAP listener and logging listener, respectively, to the complex event processing rules in these cases.
Refer to "5.4.3.3 SOAP Listener" for information on how to use the SOAP listener.
Refer to "5.4.3.4 Logging Listener" for information on how to use the logging listener.
Rule creation procedure
Design complex event processing rules in stages, without describing statements from the outset, and develop them so that the intended events will be reliably detected. The creation procedure for complex event processing rules is shown below.

Creation procedure for complex event processing rules

This section uses examples to explain the following creation procedure for complex event processing rules:

Consider what is to be achieved by using complex event processing.
Decide on what is to be achieved by using complex event processing. If this is unclear at this point, analyzing the collected events can sometimes clarify this.
Example
Example of a rule
"When someone is home, if rain is likely, recommend using the drying feature of the washing machine."
Consider the events, external data to be referenced, and output content.
Consider the events, the external data for referencing, and the output content required to create the rule. Also consider using a named window for retaining events in memory.
Example
Examples of events, external data, and output
The rule example above uses the following events, external data, and output content:
- Events
  - TV control event
    For the household appliance event data sent from the home gateway of each household, the value of the device category property is to be "Television".
  - Weather forecast event
    The forecast is to be represented by the time and weather.
- External data
  - Washing machine model information (whether it has a drying feature)
    Whether the washing machine in that household has a drying feature is to be apparent from the home gateway ID.
- Output content
  - Home gateway ID of the household to be given the recommendation
  - Recommendation details
- Named window
  - Named window for weather forecast events
    The weather forecast events for each time are to be retained for one day.

Refine the processing content.

Refine the content of the events and of the complex event processing for detecting them.

Here, "refine" is the task of using events and specific information such as complex event processing conditions and external data to make what was previously expressed in everyday language as "what is to be achieved" into a representation closer to the rule to be created.

Example

Example of refining the content of complex event processing

The table below shows the results of refining the "When someone is home, if rain is likely, recommend using the drying feature of the washing machine" rule.

Element of the rule	Refined processing content
"When someone is home"	Determine that someone is home in the household where the TV was controlled (detect a TV control notifying event).
"If rain is likely"	Reference the weather forecast information stored in the named window and, from the time of the weather forecast and the time of the TV control, check whether or not there is a forecast of rain after this time.
"Drying feature of the washing machine"	Obtain product information on the washing machine connected to the home gateway from the XTP cache.
"Recommend"	If there is a forecast of rain and if the washing machine has no drying feature, recommend hanging the clothes inside the house. If there is a forecast of rain and if the washing machine has a drying feature, recommend using it.

Create an event flowchart.
After refining the processing content is completed, summarize the event processing flow to create an event flowchart. An event flowchart associates events and their processing content in chart form. Create an event flowchart before describing the processes using complex event processing rule language, as it is useful for checking the processing content to be achieved.
Information
Legend of event flowchart
This is the legend for the event flowcharts to be used in this manual.
Example
Example of an event flowchart
This is an event flowchart for "When someone is home, if rain is likely, recommend using the drying feature of the washing machine".
1. Create a named window for retaining weather forecast events.
2. Create a Virtual Data Window (XTP cache) to see whether the washing machine has a drying feature.
3. Store weather forecast events in the weather forecast window.
4. Detect any TV control events from among the household appliance events.
5. Check the weather forecast from after the time that the TV control events occurred and leave only those events with a forecast of rain.
6. Search the washing machine feature window for the households for which a TV control was performed and for which there is a weather forecast of rain, and then add recommendation information according to whether the washing machine has a drying feature and send it to the user-developed Web service.
7. Based on the information received by the user-developed Web service, send a recommendation to the household. Control the user-developed Web service so that the same recommendation is not made to the same household twice within a fixed time period.

Create complex event processing rules.

Describe complex event processing rules that correspond to the respective elements in the event flowchart.

Example

Example of complex event processing rules that correspond to the event flowchart

Refer to Chapter 1, "Complex Event Processing Language Reference" in the Developer's Reference for information on the meanings of rules.

// 1. Create a named window for retaining weather forecast events.
@Name('WeatherWin')
create window WeatherForecastWin.std:unique(FORCASTIME).win:time(1 day)
  (FORCASTIME long, WEATHER string);
 
// 2. Create a Virtual Data Window to check whether the washing machine has a 
//    drying feature.
@VDW(cacheName='ProductFuncCache', keyProperty='gatewayId')
create window ProductFuncWin.isxtp:vdw() as (gatewayId string, dryFunc string);
 
// 3. Store weather forecast events in the named window.
@Name('InputWeather')
insert into WeatherForecastWin
  select weathfore.FORCASTIME as FORCASTIME, weathfore.WEATHER as WEATHER
    from WeatherForecastEvent as weathfore;
 
// 4. Detect TV control events.
@Name('ChkTVEvent')
insert into TVControl
  select heevnt.gatewayId as gatewayId, heevnt.updateTime as updateTime 
    from HEEvent as heevnt
    where heevnt.deviceCategory = 'Television';
 
// 5. Check the weather when TV control events occurred (detect forecast of rain).
// Use the following expression to evaluate the time zone of weather 
// forecasts. Assume FORCASTIME is "long" (milliseconds):
//   timeOfWeatherForecast
//      <= updateTimeOfTVcontrolEvent
//      <= timeOfWeatherForecast + 1 hour (3600000 milliseconds)
@Name('GetRainyEvent')
insert into TVControlRain
  select tvevnt.gatewayId as gatewayId
    from TVControl as tvevnt unidirectional, WeatherForecastWin as weather
    where updateTimeToMillis(tvevnt.updateTime) between weather.FORCASTIME 
      and (weather.FORCASTIME + 3600000) 
      and weather.WEATHER = 'rainy';
 
// 6. Check whether the washing machine has a drying feature and send the 
//    recommendation.
// Check whether ProductFuncWin dryFunc property has drying feature ("1": Yes).
// Use the SOAP listener to send the output to the application.
//   'USE_DRY_FUNC' = Recommendation ID if there is drying feature
//   'HANG_LAUNDRY_INSIDE' = Recommendation ID if there is no drying feature
// Use the logging listener to log the output.
//   table      : set log storage area.
//   properties : set property names output (output results) 
//                by delimiting with commas.
@Name('PutRecommend')
@SoapListener('soap-001')
@LoggingListener(table='/logsoap',properties='gatewayId,recommendId')
select tvevnt.gatewayId as gatewayId,
  case when product.dryFunc = '1' 
       then 'USE_DRY_FUNC' 
       else 'HANG_LAUNDRY_INSIDE'end as recommendId
  from TVControlRain as tvevnt, ProductFuncWin as product
  where product.gatewayId = tvevnt.gatewayId;

5.4.3.3 SOAP Listener

Use the SOAP listener to send the processing results of complex event processing statements to a user-developed Web service using SOAP.

To use the SOAP listener, assign the "@SoapListener" annotation in front of the complex event processing statement (SELECT statement) for which the processing results are to be sent.

Refer to "5.4.7 Designing a SOAP Listener Definition" for information on the send destination of processing results and the content of SOAP messages.

Syntax

@SoapListener("SOAPlistenerDefinitionID")
complexEventProcessingStatement(SELECT statement)

SOAP listener definition ID: Specify the development asset ID in the SOAP listener definition that defines information such as the URL that is the send destination of the Web service to which the processing results are to be sent.

Complex event processing statement (SELECT statement): Specify the complex event processing statement (SELECT statement) for which the processing results are to be sent.

5.4.3.4 Logging Listener

Use the logging listener to log the processing results of complex event processing statements in the event log using logging.

To use the logging listener, assign the "@LoggingListener" annotation in front of the complex event processing statement (SELECT statement) for which the processing results are to be logged.

Syntax

@LoggingListener(table="logStorageArea", properties="propertyNameToBeOutput")
complexEventProcessingStatement(SELECT statement)

Log storage area

Use an absolute path to specify the path in the Hadoop system in which the event log is logged.

Even if events are to be logged in the engine log of the CEP Server (if "file" is specified in the "type" element of the engine configuration file), specify a virtual path name that begins with a slash (/) (for example, /eventName) to identify the events.

Note

The "table" specification is mandatory. Even if a null value is specified, as in 'table=""', the CEP engine will start normally but logging will not be performed.

Property name to be output

Of the processing results of the SELECT statement, specify the property name to be logged. Multiple properties can also be specified if delimited using commas (,).

For a property with nested processing results, use periods (.) between the nested properties to join them.

Example

If a "child" property is nested in a "parent" property

If a "child" property is nested in a "parent" property, as shown below, specify the "child" property by describing "parent.child".

<root>
    <parent>
        <child>aaa</child>
    </parent>
</root>

Note

The "properties" specification is mandatory. Even if a null value is specified, as in 'properties=""', the CEP engine will start normally but logging will not be performed.

Complex event processing statement (SELECT statement): Specify the complex event processing statement (SELECT statement) for which the processing results are to be logged using logging.

Note

The logging to be defined in an event type definition and the logging to be defined in a rule definition (logging listener) can have separate log storage areas as output destinations. Note, however, that logging to a different Hadoop system is impossible.
If the value of the output property is "null", this is converted to a blank space before being output to the event log.
If the output property is a numeric item, this undergoes string conversion before being output to the event log.
If an output property name that does not exist is specified, a blank space is output to the event log.
If there are double quote marks (") in data, these will be output in duplicate within double quote marks within the data.
Output example of 'aa"bb"aa':
```
"aa""bb""aa"
```