The Usage DB Environment Definition File defines the conditions for creating databases used to store various log data on the Web services. The file also defines the conditions for creating usage DBs.
Analysis window data and CSV output data are all supplied from the usage DB.
The Usage DB Environment Definition File is a text file. Use a text editor such as Notepad to create and edit the file.
[Windows]
<Variable file storage directory>\control\DatabaseConfig |
[UNIX]
/etc/opt/FJSVssqc/DatabaseConfig |
A sample Usage DB Environment Definition File is stored in the folder shown below during installation. The user can edit the sample file to create a Usage DB Environment Definition File.
[Windows]
<Installation directory>\sample\DatabaseConfig |
[UNIX]
/opt/FJSVssqc/sample/DatabaseConfig |
Character codes of the text are as follows:
[Windows]
ASCII |
[UNIX]
ASCII |
In the Usage DB Environment Definition File, time stamps are provided on the first and last lines for validity checking, and the definitions for the server(s) to be analyzed are provided between the time stamps.
Up to 20 definitions can be provided for the server(s) to be analyzed.
The definition for one analysis target server consists of the following definition blocks:
Analysis target server definition block * 1
Analysis target log definition block * 1 or 2
The whole file structure is shown below:
Time stamp Analysis target server definition block Analysis target log definition block Analysis target log definition block : Analysis target server definition block Analysis target log definition block : Time stamp |
Note
The analysis target server definition is a definition made for a server to be analyzed.
If two or more servers are to be analyzed, the corresponding number of definitions should be provided.
The analysis target log definition is a definition about the log for the server defined by the analysis target server definition.
One or two analysis target log definitions can be made for one analysis target server definition.
If two log definitions are provided, the following restrictions apply to their combination:
Analysis target log definition 1 | Analysis target log definition 2 | Validity | Remarks |
|---|---|---|---|
Web server or Proxy server log | SQC extended log | Valid | None |
Web server or Proxy server log | Web server or Proxy server log | Not valid | If two Web server or Proxy server log definitions are provided, separate the analysis target server definition for each of the log definitions. In this case, analysis target log definition 1 is valid. |
SQC extended log | SQC extended log | Not valid | If two SQC extended log definitions are provided, separate the analysis target server definition for each of the log definitions. In this case, analysis target log definition 1 is valid. |
The coding format of the Usage DB Environment Definition File is shown below:
Line coding format | Description |
|---|---|
[xxxxxxxxxx] | Indicates the beginning of a definition block as well as the end of the previous definition block |
xxxx = xxxxxxxxxx | Indicates parameters in a definition block. '|' means OR, i.e., the parameter on either side can be specified. |
none | A null line is treated as a comment. |
# xxxxxxxxxx | A line beginning with '#' is treated as a comment. |
In the Usage DB Environment Definition File, an analysis target server definition block should be provided in the following format:
Format
[Server] Symbol = server-symbol Name = server-name Type = web | proxy Domain = domain-name DatabaseInterval = day | week(sun|mon|tue|wed|thu|fri|sat) | month(1-31) SearchDNS = yes | no RequestURLSuffix = "suffix-list" DatabaseMode = db | csv | both DefaultURLPage = "default-url" |
Explanation
[Server]
Indicates the beginning of an analysis target server definition block.
Symbol
Defines the symbol of the analysis target server.
For server-symbol, specify a symbol with up to 10 alphanumeric characters (a to z, A to Z, and 0 to 9) beginning with an alphabetic character (a to z or A to Z).
This symbol is used as the directory name when a directory corresponding to the analysis target server is created under the usage DB storage directory.
Name
Defines the name of the analysis target server.
For server-name, specify the server name with up to 20 characters.
The specified name is displayed in the analysis target server box in the analysis window and used as the keyword for selecting the analysis target server. If the specified name is too long to be accommodated in the analysis target server box, the portion that is able to accommodate from the beginning is displayed. If multiple analysis target servers are specified, specify their names so that they can be distinguished by the leading parts of the names.
Type
Defines the type of the analysis target server.
The meanings of the options are as follows:
Option | Meaning |
|---|---|
web | Web server |
proxy | Proxy server |
Domain
Defines the domain name of the analysis target server.
For domain-name, specify the domain name (including a subdomain name). For instance, when the fully qualified domain name (FQDN) is host.xxxx.yyyy.zzzz, specify xxxx.yyyy.zzzz.
In the analysis window, detailed data by the host name can be displayed. The host name refers to the FQDN. The domain name specified here is used for conversion to FQDN if the host name in the analysis target server log is recorded only with the name in the domain.
DatabaseInterval
Defines the changeover unit for the usage DB corresponding to the analysis target server.
The meanings of the options are as follows:
Option | Meaning |
|---|---|
day | Daily |
week(...) | Weekly |
month(...) | Monthly |
If week is specified, specify also one of the following values to indicate the start day of the week in parentheses:
Option | Meaning |
|---|---|
sun | Sunday |
mon | Monday |
tue | Tuesday |
wed | Wednesday |
thu | Thursday |
fri | Friday |
sat | Saturday |
If month is specified, specify also one of the following values to indicate the start day of the month. If the specified day does not exist, the last day of the month is used.
Option | Meaning |
|---|---|
1 to 31 | Day of the month |
The default is shown below. If the default is to be used, the line itself can be omitted.
DatabaseInterval = week(sun)
See
Refer to "11.1.1.7 Usage DB switching" for details of usage DB changeover.
SearchDNS
Defines whether to perform DNS search.
In the analysis window, detailed data by the host name and by the IP address can be displayed. However, only one of them is recorded in the analysis target server log. This parameter specifies whether to perform DNS search to retrieve the other. The meanings of the options are as follows:
Option | Meaning |
|---|---|
yes | Perform DNS search. |
no | Do not perform DNS search. |
The default is shown below. If the default is to be used, the line itself can be omitted.
SearchDNS = yes
Note
If SearchDNS = no is specified, the usage DB registration engine does not perform DNS search and performs analysis based on the record in the analysis target log file. This affects the display in the analysis window as follows:
When the client names in the analysis target log file are recorded with their IP addresses
When one of the following items is specified in the analysis method box or advanced analysis box, data for the corresponding FQDN is analyzed and displayed:
For each client host name
For each referrer host name
For each remote host name
When the client names in the analysis target log file are recorded with their host names
When one of the following items is specified in the analysis method box or advanced analysis box, data for the corresponding FQDN is analyzed and displayed:
For each client IP address
For each referrer IP address
For each remote IP address
RequestURLSuffix
Defines URLs subject to request analysis.
To check clients' interests, it is important that request analysis is performed from the viewpoint of client's will. Requests generated by clients' will tend to be targeted to URLs having specific extensions such as html. For suffix-list, specify the URL extensions corresponding to clients' will by separating them with a comma (,). Up to 40 URL extensions can be specified.
The default is shown below. If the default is to be used, the line itself can be omitted.
RequestURLSuffix = "html,htm,shtml,shtm,stm,cgi,asp,pl,tcl,sh"
Note
The user can drill down according to URL by selecting detail by URL extension in the analysis method box. In this case, if the extension for drilldown is not specified for RequestURLSuffix in the Usage DB Environment Definition File, the extension is treated as excluded for analysis. "No data is available for this period" is displayed.
DatabaseMode
Defines the usage DB operation mode.
This definition can be specified only on a managed server (Agent for Business). It is ignored if specified on the Management Server (Manager). The meanings of the options are as follows:
Option | Meaning |
|---|---|
db | Only the usage DB is used. Specify this option if analysis is performed on the managed server. |
csv | Only the Usage Log file is used. The CSV format file is transferred to the Management Server and stored in the usage DB of the Management Server. Specify this option if analysis is performed on the Management Server. |
both | Both the usage DB and CS format log file are used. A Usage Log file is created as well as a usage DB. |
The default is shown below. If the default is to be used, the line itself can be omitted.
DatabaseMode = db
Note
If DatabaseMode = csv is specified, log data for the analysis server is transferred to the Management Server and therefore analysis cannot be performed on the managed server.
DefaultURLPage
Defines the URL file that is automatically added if the path information file name is omitted in the analysis target log.
When a Web site is accessed, the file name may be omitted in the URL specification such as http://www.fujitsu.com/. If so, the Web server can access the Web content by automatically adding the default file name.
By default the Trend Viewer performs analysis separately when an access is made with the file name specified and when an access is made with the file name omitted. If this parameter is defined, the defined file name is automatically added even when an access is made with the file name omitted, thereby enabling the same analysis to be performed as when an access is made with the file name specified.
For default-url, specify the file name to be automatically added if the file name is omitted in the URL specification. Observe the following specification rules:
The file name must not exceed 255 characters (bytes).
The file name must be enclosed in double quotation marks.
Alphanumeric characters excluding blanks and the following characters can be used for the file name:
" # % , < > \ |
Only one file name can be specified.
This parameter has no default definition and is valid only when it is specified.
Example
A sample definition is as shown below:
[Server] Symbol = PUBLIC Name = OpenServer Type = web Domain = xxxx.yyyy.zzzz DatabaseInterval = week(sun) SearchDNS = yes RequestURLSuffix = "html,htm,shtml,shtm,stm,cgi,asp,pl,tcl,sh" DatabaseMode = db DefaultURLPage = "index.html" |
In the Usage DB Environment Definition File, an analysis target log definition block should be provided in the following format:
Format
[Log] Symbol = log-symbol Name = log-name Path = log-path Format = format-symbol | "format" Region = diff-time |
Explanation
[Log]
Indicates the beginning of an analysis target log definition block.
Symbol
Defines the symbol of the analysis target log.
For log-symbol, specify a symbol with up to 10 alphanumeric characters (a to z, A to Z, and 0 to 9) beginning with an alphabetic character (a to z or A to Z).
Name
Defines the name of the analysis target log.
For server-name, specify the log name with up to 20 characters.
Path
Defines the path of the analysis target log file.
For log-path, specify the absolute path of the file. If multiple log files are created under the same directory, use the wildcard (*: any characters numbering 0 or more) to specify all of these files. If the path contains any blanks, enclose the entire path within double quotation marks(").
Format
Defines the record format in the analysis target log file.
Specify format-symbol or "format" for the record format.
If format-symbol is used, specify a symbol corresponding to a fixed format. For symbols that can be specified, refer to Kind and meaning of symbol" later in this section.
If "format" is used, specify tokens corresponding to data along with the delimiters by placing them in a line. If the record format in the analysis target log file does not correspond to any fixed format, use "Kind and meaning of token" for specification.
To perform analysis of the managed server log on the Management Server, specify SQC-CSV using format-symbol as the record format of the relevant managed server on the Management Server side.
Point
The usage DB registration engine treats all characters, which do not match the token string specified for "format", as delimiters. Note that any misspelled token is treaded as a delimiter.
The usage DB registration engine reads logs from the analysis target log file and processes only those logs whose record formats match the formats specified here, and then stores them in the usage DB. Log records whose record format does not match any format specified here are not processed.
Thus, logs in the specified record format need to be output to the log file to be analyzed. To draw attention to any possible error in the record formats specified here, the usage DB registration engine stops processing if a given number of records whose record format does not match the format specified here are found from the start of the log file.
The token types and their meanings are as follows:
Kind and meaning of token
Token | Meaning |
|---|---|
c-host | Host name of IP address of the client |
c-user | Authentication user name of the client |
s-time{time-format} | Time the server completed request processing |
c-request (*1) | First request sent from the client to the server |
s-method (*1) | Method requested by the client to the server (part of c-request) |
s-url (*1) | URL requested by the client to the server (part of -request) |
s-host (*1) | Host name of IP address requested by the client to the server (part of s-url) |
s-path (*1) | File path requested by the client to the server (part of s-url) |
s-status | Status code sent from the server to the client |
r-status | Status code sent from the remote server to the server |
s-bytes | Number of bytes transferred from the server to the client |
r-bytes | Number of bytes transferred from the remote server to the server |
l-url | Content of the Refer request header sent from the client to the server |
c-agent | Content of the User-Agent request header sent from the client to the server |
s-elapse{elapse-format} | Time required by the server for request processing |
c-cookie{cookie-format} | Content of Cookie sent from the client to the server |
* | Variable element other than the above |
\ | Escape character (To specify ["] [\], add escape characters like [\"] [\\].) |
Point
(*1) The following shows the relationships among c-request, s-method, s-url, s-host, and s-path:
In time-format, place the tokens corresponding to the time format along with the delimiters in a line. The following table lists the meanings of time-format tokens:
Token | Meaning |
|---|---|
yyyy | Year (1980 to 2038) |
yy | Year (00 to 99) |
mm | Month (01 to 12) |
mon | Month (Jan, Feb, Mar, Apr, May, Jun, Jul, Aug, Sep, Oct, Nov, Dec) |
month | Month (January, February, March, April, May, June, July, August, September, October, November, December) |
dd | Day (01 to 31) |
HH | Hour (00 to 23) |
MM | Minute (00 to 59) |
SS | Second (00 to 59) |
seconds | Total seconds |
In elapse-format, describe the token indicating the unit of the elapsed time. The token is one of the following:
Token | Meaning |
|---|---|
s | Seconds |
ms | Milliseconds |
In cookie-format, place tokens corresponding to the cookie format along with the delimiters in a line. The following table lists the meanings of the cookie-format tokens:
Token | Meaning |
|---|---|
id | Access ID defined in Cookie |
* | Variable elements other than the above |
Note
If the record format "format" is defined by listing the tokens, make sure that the following required tokens have been specified. Note that the analysis cannot be performed if these required tokens are not specified.
Required tokens |
|---|
s-time |
c-host |
s-url (or c-request, s-path) |
s-status |
s-bytes |
For Microsoft Internet Information Services, the default log file format does not contain log information corresponding to the "s-bytes" required token (the number of bytes transferred from the server to the client). Make sure this information is defined in the log file format before performing analysis. (The equivalent for Microsoft Internet Information Services is "sc-bytes".)
Point
If the record format is defined using "format", make sure the required tokens are incuded.
Analysis to be conducted | Require token |
|---|---|
For each authentication user | c-user |
For each referrer | l-url |
For each browser | c-agent |
Request processing time on the server | s-elapse |
Error and cache data on the remote server as well in proxy server analysis | r-status |
The following table lists the types of symbols (when web is defined in the analysis target server definition) for the Web server and their meanings:
Kind and meaning of symbol
Symbol | Corresponding log |
|---|---|
"format" content | |
Common | Common Logfile Format of W3C. This format corresponds to the following log formats:
|
"c-host * c-user [s-time{dd/mon/yyyy:HH:MM:SS} *] \"c-request\" s-status s-bytes" | |
Common+R+U | The Refer request header and User-Agent request header contents are added to Common. This format can be adapted to the following formats or their customized formats:
|
"c-host * c-user [s-time{dd/mon/yyyy:HH:MM:SS} *] \"c-request\" s-status s-bytes \"l-url\" \"c-agent\"" | |
Microsoft-MS | Microsoft Internet Information Server's specific format. This format corresponds to the following log format:
*1 Microsoft Internet Information Services 5.0 has a different format to V3.0 and V4.0. Here, specify the format directly by placing tokens. |
"c-host, c-user, s-time{yy/mm/dd, HH:MM:SS}, *, *, *, *, *, s-bytes, s-status, *, s-method, s-path, *" | |
SQC-Extend | SQC extended log format |
* Symbol specification only | |
SQC-CSV | Usage Log file of Systemwalker Service Quality Coordinator To analyze a log of the managed server on the Management Server, specify SQC-CSV for the relevant managed server. |
* Symbol specification only | |
SQC-ExtCSV | CSV format extended log file of Systemwalker Service Quality Coordinator To analyze a response log of the managed server on the Management Server, specify SQC-ExtCSV for the relevant managed server. |
* Symbol specification only |
Note
When specifying the record format using a symbol, compare the content of the "format" corresponding to this symbol with a log that has actually been taken and make sure that the formats match before making the specification. Take particular care with formats that have date sections, as this may vary depending on the system.
The following table lists the types of symbols (when proxy is defined in the analysis target server definition) for the Proxy server and their meanings:
Symbol | Corresponding log |
|---|---|
"format" content | |
Common | "Common Logfile Format" of W3C. This format corresponds to the following log formats:
|
"c-host * c-user [s-time{dd/mon/yyyy:HH:MM:SS} *] \"c-request\" s-status s-bytes" | |
Common+R+U | The Refer request header and User-Agent request header contents are added to Common. This format can be adapted to the following formats or their customized formats:
|
"c-host * c-user [s-time{dd/mon/yyyy:HH:MM:SS} *] \"c-request\" s-status s-bytes \"l-url\" \"c-agent\"" | |
Common+Ts | The processing time (seconds) is added to Common. This format can be adapted to the following formats or their customized formats:
|
"c-host * c-user [s-time{dd/mon/yyyy:HH:MM:SS} *] \"c-request\" s-status s-bytes s-elapse{s}" | |
Common+Tms | The processing time (milliseconds) is added to Common. This format can be adapted to the following formats or their customized formats:
|
"c-host * c-user [s-time{dd/mon/yyyy:HH:MM:SS} *] \"c-request\" s-status s-bytes s-elapse{ms}" | |
Netscape-Extend | Netscape Proxy Server's specific format. This format corresponds to the following log formats: Extended log format and Extended2 log format of Netscape Proxy Server |
"c-host * c-user [s-time{dd/mon/yyyy:HH:MM:SS} *] \"c-request\" s-status s-bytes r-status r-bytes * * * * * * s-elapse{s}" | |
Squid-Native11 | Squid's specific format. This format corresponds to the following log format:
|
"s-time{seconds} s-elapse{ms} c-host */s-status s-bytes s-method s-url * */* *" | |
Microsoft-Native | Microsoft Proxy Server's specific format. This format corresponds to the following log format:
|
"c-host, c-user, c-agent, *, time{yy/mm/dd, HH:MM:SS}, *, *, *, *, *, *, s-elapse{ms}, s-bytes, r-bytes, *, *, s-method, s-url, *, *, s-status, *" | |
DeleGate-Default | DeleGate's specific format. This format corresponds to the following log format:
|
"c-host * c-user [s-time{dd/mon/yyyy:HH:MM:SS} *] \"c-request\" s-status s-bytes :*" | |
InfoProxy-Extend | Fujitsu InfoProxy's specific format. This format corresponds to the following log format:
|
"c-host * c-user [s-time{dd/mon/yyyy:HH:MM:SS} *] \"c-request\" s-status s-bytes s-elapse{ms} r-status r-bytes * * * * * * * * * * * * *" | |
SQC-CSV | Usage Log file of Systemwalker Service Quality Coordinator To analyze a log of the managed server on the Management Server, specify SQC-CSV for the relevant managed server. |
* Symbol specification only |
Note
When specifying the record format using a symbol, compare the content of the "format" corresponding to this symbol with a log that has actually been taken and make sure that the formats match before making the specification. Take particular care with formats that have date sections, as this may vary depending on the system.
When the log record format is specified using a symbol for "format", the analysis methods that can be used are as follows:
No | Symbol value | Analysis type |
|---|---|---|
1 | Common |
|
2 | Common+R+U |
|
3 | Microsoft-MS |
|
4 | Common+Ts |
|
5 | Common+Tms |
Note: In "Request report", "Traffic report", and "Cache report", the elapsed time indicating the processing capability of the Web server or Proxy server to be analyzed is in milliseconds. The elapsed time is displayed on each analysis window. |
6 | Netscape-Extend |
Note: In "Request report", "Traffic report", and "Cache report", the elapsed time indicating the processing capability of the Web server or Proxy server to be analyzed is in milliseconds. The elapsed time is displayed on each analysis window. |
7 | Squid-Native11 |
Note: In "Request report", "Traffic report", and "Cache report", the elapsed time indicating the processing capability of the Web server or Proxy server to be analyzed is in milliseconds. The elapsed time is displayed on each analysis window. |
8 | Microsoft-Native |
Note: In "Request report", "Traffic report", and "Cache report", the elapsed time indicating the processing capability of the Web server or Proxy server to be analyzed is in milliseconds. The elapsed time is displayed on each analysis window. |
9 | DeleGate-Default |
|
10 | InfoProxy-Extend |
Note: In "Request report", "Traffic report", and "Cache report", the elapsed time indicating the processing capability of the Web server or Proxy server to be analyzed is in milliseconds. The elapsed time is displayed on each analysis window. |
11 | SQC-Extend |
|
Region
Defines a region for the time data recorded in the analysis target log file.
For diff-time, specify the region time using the time difference with GMT (Greenwich Mean Time). The specification format is shown below:
Format | Explanation |
|---|---|
[+|-]HHMM | +: Fast -: Slow HH: Hour (00 to 23) MM: Minute (00 to 59) |
Note
Regions of time data recorded in the log depend on the type of the Web server or Proxy server to be analyzed.
It is necessary to check which format of time data is collected. For information about the format of collected time data, refer to the manual of each server.
Example
Sample definitions are shown below:
[Windows]
[Log] Symbol = WWW Name = WebServerLog # W3C Extended log file format # Option: Date, time, client IP address, method, URI Stem, Http status, and number of transmitted bytes Path = "C:\WINNT\system32\LogFiles\W3SVC1\ex*.log" Format = "s-time{yyyy-mm-dd HH:MM:SS} c-host s-method s-url s-status s-bytes" Region = +0000 |
[UNIX]
[Log] Symbol = WWW Name = WebServerLog Path = /usr/local/apache/logs/access_log Format = Common Region = +0900 |
