This section describes the CSV log files created by the Usage DB Registration Engine on the Managed Server as a means of transferring usage data to the Management Server.
This section explains the format of the CSV format log file.
CSV format log files are used for Web server and proxy server logging. In addition, CSV format extended log files are used for analysis of response reports.
Storage location and file name
CSV format log files are stored in the following directory:
|
* < > indicates a directory.
*1 The server directory is assigned the name specified by Symbol in the analysis target server definition block of the Usage DB Environment Definition File.
*2 The variable (yyyymmdd_nnn) in the CSV format log file name is shown below. The date means the date on which the usage DB was switched (created).
yyyy | Year (1980 and after) |
mm | Month (01 to 12) |
dd | Day (01 to 31) |
nnn | Serial number (001 to 999) |
When response report analysis is performed, a CSV format extended log file for response report analysis is created with the following file name under the server directory shown above. The variable (yyyymmdd_nn) in the name of the CSV format extended log file is the same as that in the CSV format log file name.
eyyyymmdd_nnn.csv
In addition, an index file is created with the following file name under the server directory. This is to improve the efficiency of access to the CSV format file when the CSV format log file is transferred from the managed server to the Management Server. The index file is a special file for the Usage DB Registration Engine. Do not use the index file.
dbyyyymmdd.index
eyyyymmdd.index
File format
The CSV format log file and CSV format extended log file are text files. The character codes used for the texts vary depending on the platform.
[Windows]
Character code |
|---|
ASCII |
[UNIX]
Character code |
|---|
ASCII |
Data in the file is separated by a comma (,). The file consists of the following lines.
Title line Data line Data line : : Data line |
The following table explains each line.
Line | Explanation |
|---|---|
Title | The field name (item name) of each field is output to only the first line as the title line. Each field is delimited by a comma (,). |
Data | Data lines are output on the second and subsequent lines. One valid record in the Web server log is output as one data line. Data is output in ascending order (old to new) of date and time. Each field is separated with a comma (,). A hyphen (-) is output in fields for which there is no applicable data. |
Format of each line
The following table lists details of the title line and data line for each data type. Although the format description in this table extends over multiple lines, the actual output is one line.
CSV format log file
Line | File output format |
|---|---|
Title line (first line) | server,s-date,s-time,c-host,c-ip,s-method,s-protocol,s-host,s-ip,s-path,s-status,r-status,s-byte,r-byte,l-url,c-agent,s-elaps,c-cookie-id,c-user |
data line (from the second line) | "server","yyyy/mm/dd","hh:mm:ss","c-host","c-ip","s-method","s-protocol","s-host","s-ip","s-path","s-status","r-status","s-byte","r-byte","l-url","c-agent","s-elaps","c-cookie-id","c-user" |
CSV format extended log file
Line | File output format |
|---|---|
Title line (first line) | server,s-date,s-time,c-host,s-url,c-response |
Data line (from second line) | "server","yyyy/mm/dd","hh:mm:ss","c-host","c-ip","s-url","c-response" |
Explanation of data line fields (second and subsequent lines)
CSV format log file
Field | Meaning |
|---|---|
"server" | Web server name |
"yyyy/mm/dd" | Date on which the server completes processing of a request
|
"hh:mm:ss" | Time at which the server completes processing of a request
|
"c-host" | Client host name |
"c-ip" | Client IP address |
"s-method" | Method the client requests from the server |
"s-protocol" | Protocol the client requests from the server |
"s-host" | Host name the client requests from the server |
"s-ip" | IP address the client requests from the server |
"s-path" | File path the client requests from the server |
"s-status" | Status code sent to the client by the server |
"r-status" | Status code sent to the server by the remote server |
"s-byte" | Number of bytes sent by the server to the client |
"r-byte" | Number of bytes sent by the remote server to the server |
"l-url" | Referrer URL |
"c-agent" | Content of the User-Agent request header sent by the client to the server |
"s-elaps" | Time period required by the server for request processing (in milliseconds) |
"c-cookie-id" | Access ID defined for a cookie |
"c-user" | Client authentication user name |
CSV format log file
Field | Meaning |
|---|---|
"server" | Web server name |
"yyyy/mm/dd" | Date on which the server completes processing of a request
|
"hh:mm:ss" | Time at which the server completes processing of a request
|
"c-host" | Client host name |
"s-url" | Path of the HTML document that the client requested of the server |
"c-response" | Response time in milliseconds |
An example for calculating an approximate capacity under the following conditions is shown below. Refer to the example to estimate the capacity considering the period in which the CSV-format log file is output.
Conditions
Data type | Visit information |
|---|---|
Size per line | About 250 bytes (depending on the length of the client name and URL) |
Access count | About 10,000/day |
Approximate maximum size per day
About 250 bytes * about 10,000 = about 2.5 MB (/day)
The Usage Log file is created when log data of the Managed Server is transferred to the Management Server and analyzed on the same Server. The file is no longer needed after it is transferred to the Management Server and stored in the usage DB.
This section explains how to delete the Usage Log file that is no longer needed after it is stored in the usage DB on the Management Server. Before deleting the Usage Log file, make a backup copy of the file as needed in preparation for problems such as destruction of the usage DB.
Usage Log file that can be deleted
Usage Log file that can be deleted
The Usage Log files that were transferred to the Management Server and stored in the usage DB can be deleted.
Furthermore, the Usage Log files stored in the usage DB of the Usage Log files on the Managed Server and the Management Server can be deleted.
The files to be deleted are as follows:
dbyyyymmdd_nnn.csv
eyyyymmdd_nnn.csv
dbyyyymmdd.index
eyyyymmdd.index
Note
"dbyyyymmdd.index" and "eyyyymmdd.index" are stored on the managed server only.
The storage locations for the files to be deleted are as follows:
[Windows]
<Variable file storage directory>\database\csv\Server directory\ |
<Variable file storage directory>\database\csv\Server directory\Server_Csv_Backup\ |
<Variable file storage directory>\database\csv\Server directory\Extend_Csv_Backup\ |
[UNIX]
/var/opt/FJSVssqc/database/csv/Server directory/ |
/var/opt/FJSVssqc/database/csv/Server directory/Server_Csv_Backup/ |
/var/opt/FJSVssqc/database/csv/Server directory/Extend_Csv_Backup/ |
Note
"Server directory" is the name specified by "Symbol" in the analysis target server definition block in the usage database environment definition file.
Conditions for deleting CSV format log files
Delete only the CSV format log files that meet the following conditions.
Management server side
Use a text editor to open the log files stored in the following directory. If there are no errors in any of the analysis target servers, delete those files that are stored in the directory for storing CSV format log files and whose dates are more than three days (server actual operating day) old counting back from the date of the latest file.
Note
Do not delete any files if there are errors.
[Windows]
<Variable file storage directory>\log\dbreg.log |
[UNIX]
/var/opt/FJSVssqc/log/dbreg.log |
Managed server side
Check the files that have been transferred within the directory for storing CSV format log files, and delete those files whose dates are more than three days (server actual operating day) old counting back from the date of the latest file.
See
Refer to "Storage location and file name" in "15.2.6.1 Usage Log file format" for information on how to read the dates for CSV format log files.
Note
Make sure that the following processes are not running when CSV format log files are deleted.
[Windows]
dbregmng.exe |
dbreg.exe |
[UNIX]
dbregmng |
dbreg |
Automatically deleting an unnecessary Usage Log file
The user may want to automatically delete the Usage Log file that has already been transferred to the Management Server and stored in the usage DB.
To automatically delete an unnecessary Usage Log file, create a batch file (or shell script) used for deletion and run it using the scheduler function.
For the scheduler function, you can use the OS standard function as shown below or introduce and use the product with the scheduler function such as Systemwalker Operation Manager.
OS standard scheduler function
[Windows]
AT command
[UNIX]
cron command
