2.4 Recovery Errors

Interstage Shunsaku Data Manager Troubleshooting - Microsoft(R) Windows(R) 2000/ Microsoft(R) Windows Server(TM) 2003 - - UNIX -

Contents Index

Chapter 2 Examples of Troubleshooting

2.4 Recovery Errors

The following are examples of recovery errors:

shn30318u: An I/O error with the director data file.
shn30328u: Invalid backup data error.
shn30331u: An error in the recovery procedure.
shn30332u: An insufficient space error for the director data file.
shn30335u: An I/O error in the operation log file.
shn30338u: Not the latest backup data.
shn30341w: Discontinuous update log data error.
shn30355u: A character code error.
shn30364u: A searcher error.

An I/O error with the director data file

Error Event

After the shundrecover command is executed, the following message is output:

shn30318u: An I/O error has occurred with a director data file.errno= d*  [shund1]

d*: Error number for the system call

Error Confirmation

The status of the director is "RECOVER". Also, the status of the director data file is "IOERROR". Therefore, data cannot be searched or and updated through the application.

shundstate -s shund1
Shunsaku shundstate 2006/09/01 13:00:00

Time     State                DataFileStatus       DataSize             Fragments(%)
         LastBackedUp         OperationLogStatus   OperationLogSize
         Records              ReadRecords          ReadTime(sec)        ReadSize

13:00:00 RECOVER              IOERROR             9.67MB               66
         2006/08/31 22:00:00  NORMAL               20.8MB
         10000                10000                0.829                9.78MB

Corrective Action

Replace the disk and execute the following recovery procedure:

Replace the disk where the director data file is located.
If the operating system has to be restarted during disk replacement, stop the Shunsaku system and restart the system again after replacing the disk.
Execute the shundstate command to confirm the status of the director and director data file.

If the status of the director is "ACTIVE", execute the shundrecover command to declare the start of recovery:

shundrecover -s shund1 -b
shun: INFO: shn21001i: Starting process... [shund1]
shun: INFO: shn30327i: Recovery start declaration processing has been completed. [shund1]
shun: INFO: shn21002i: Processing has been completed. [shund1]

Restore the backup of the director data file again.
Execute the shundrecover command to recovery the system.

The following example is of a message that shows the recovery to its latest state:

shundrecover -s shund1 -e
shun: INFO: shn21001i: Starting process... [shund1]
shun: INFO: shn30334i: Recovery end declaration processing is complete. [shund1]
shun: INFO: shn21002i: Processing has been completed. [shund1]

Execute the shundstate command to confirm whether the status of the director is "ACTIVE" and the status of the director data file is "NORMAL":

shundstate -s shund1
Shunsaku shundstate 2006/09/01 13:10:00

Time     State                DataFileStatus       DataSize             Fragments(%)
         LastBackedUp         OperationLogStatus   OperationLogSize
         Records              ReadRecords          ReadTime(sec)        ReadSize

13:10:00 ACTIVE               NORMAL                9.67MB               66
         2006/08/31 22:00:00  NORMAL                20.8MB
         10000                10000                 0.829                9.78MB

Execute the shundbackup command to back up the director data file.
Recommence operations.

Invalid backup data error

Error Event

After the shundrecover command to declare the end of recovery is executed, the following message is output:

shn30328u: There is an error with the backup data that has been restored. [shund1]

Cause

There is an error in the restored director data file backup data.

The director data file backup data consists of the following file types:

If any part of the above files is absent or if backup data from other backup point exists, the error message (shn30328u) is output if the shundrecover command is executed with the -e option to declare the end of recovery.

Corrective Action

Restore the correct backup data and re-execute the shundrecover with the -e option to declare the end of recovery.

If the status of the director displayed by the shundstate command is "ACTIVE", declare the start of recovery. This is not necessary if the status of the director is "RECOVER".

shundrecover -s shund1 -b
shun: INFO: shn21001i: Starting process... [shund1]
shun: INFO: shn30327i: Recovery start declaration processing has been completed. [shund1]
shun: INFO: shn21002i: Processing has been completed. [shund1]

Restore the correct backup data.

Execute the shundrecover command to declare the end of recovery. The following example is of a message that shows the recovery to its latest state:

shundrecover -s shund1 -e
shun: INFO: shn21001i: Starting process... [shund1]
shun: INFO: shn30334i: Recovery end declaration processing is complete. [shund1]
shun: INFO: shn21002i: Processing has been completed. [shund1]

Confirm whether the status of the director displayed by the shundstate command is "ACTIVE":

shundstate -s shund1
Shunsaku shundstate 2006/09/01 13:10:00

Time     State                DataFileStatus       DataSize             Fragments(%)
         LastBackedUp         OperationLogStatus   OperationLogSize
         Records              ReadRecords          ReadTime(sec)        ReadSize

13:10:00 ACTIVE               NORMAL                9.67MB               66
         2006/08/31 22:00:00  NORMAL                20.8MB
         10000                10000                 0.829                9.78MB

Execute the shundbackup command to back up the director data file.
Recommence operations.

Notes

The shundrecover command can only be used to recover the director data file backup data collected by the shundbackup command.
If the director data file is backed up when the Shunsaku system is stopped, follow the procedure below. However, only data from the backup point can be restored even if the operation log file is used.
After the recovery, execute the shundbackup command to back up the director data file.

The following procedure restores data backed up while the Shunsaku system was stopped:

Stop the Shunsaku system.

shunsysstop -n shunsaku (Shunsaku system name)
shn30579i:Stopping Shunsaku system. [shunsaku]
shn30582i:Stopping conductor. [shunsaku]
shn30583i:Stopping sorter. [shunsaku]
shn30584i:Stopping director. [shunsaku]
shn30580i:Shunsaku system has been stopped. [shunsaku]

Restore the backup data collected while the Shunsaku system was stopped.

Start the Shunsaku system

shunsystart -n shunsaku (Shunsaku system name)
shn30567i:Activating Shunsaku system. [shunsaku]
shn30573i:Activating director. [shunsaku]
shn30572i:Activating sorter. [shunsaku]
shn30571i:Activating conductor. [shunsaku]

Confirm whether the status of the director displayed by shundstate command is "ACTIVE":

shundstate -s shund1
Shunsaku shundstate 2006/09/01 13:10:00

Time     State                DataFileStatus       DataSize             Fragments(%)
         LastBackedUp         OperationLogStatus   OperationLogSize
         Records              ReadRecords          ReadTime(sec)        ReadSize

13:10:00 ACTIVE               NORMAL                9.67MB               66
         2006/08/31 22:00:00  NORMAL                20.8MB
         10000                10000                 0.829                9.78MB

Execute the shundbackup command to back up the director data file.
Recommence operations.

An error in the recovery procedure

Error Event

After the shundrecover command to declare the end of recovery is executed, a recovery error may occur and the following message may be output:

shn30331u: The backup data are not restored in director data folder, or director data file is invalid. [shund1]

Cause

The following three causes are possible:

Cause 1

Data was backed up after the shundrecover command was executed with the -b option but before the recovery end declared by the -e option and this data has not been restored.

The error below is output if the back up data was not restored at the correct time during the recovery procedure.

[An example backup data being restored at the incorrect time]

Cause 2

The shundrecover command was executed with the -e option to declare the end of recovery, however the backup data was not restored and the error occurs.

The backup data is written to the director data file when the end of the recovery is declared by executing the shundrecover command with the -e option. If an error occurs during the recovery, the director data file will not be able to stop the process. Therefore, it is necessary to restore the backup data again.

Cause 3

The backup data restored is not the backup data collected by the shundbackup command.

The recovery with the shundrecover command can only be used with the director data file back up data that was backed up after the recovery start declaration and before the recovery end declaration collected by the shundbackup command with the -b and -e options specified.
A recovery error will occur and this message will be output if backup data collected from other methods is used.

Corrective Action

Execute the following recovery procedure:

If the status of the director displayed by the shundstate command is "ACTIVE", declare the start of recovery. This is not necessary if the status of the director is "RECOVER".

shundrecover -s shund1 -b
shun: INFO: shn21001i: Starting process... [shund1]
shun: INFO: shn30327i: Recovery start declaration processing has been completed. [shund1]
shun: INFO: shn21002i: Processing has been completed. [shund1]

Restore the backup data of the director data file:
Restore the correct backup data that has been collected between the -b and the -e option of the shundbackup command.
Execute the shundrecover command to declare the end of recovery:

The following example is of a message showing the recovery to its latest state:

shundrecover -s shund1 -e
shun: INFO: shn21001i: Starting process... [shund1]
shun: INFO: shn30334i: Recovery end declaration processing is complete. [shund1]
shun: INFO: shn21002i: Processing has been completed. [shund1]

Confirm whether the status of the director displayed by the shundstate command is "ACTIVE":

shundstate -s shund1
Shunsaku shundstate 2006/09/01 13:10:00

Time     State                DataFileStatus       DataSize             Fragments(%)
         LastBackedUp         OperationLogStatus   OperationLogSize
         Records              ReadRecords          ReadTime(sec)        ReadSize

13:10:00 ACTIVE               NORMAL                9.67MB               66
         2006/08/31 22:00:00  NORMAL                20.8MB
         10000                10000                 0.829                9.78MB

Execute the shundbackup command to back up the director data file.
Recommence operations.

An insufficient space error for the director data file

Error Event

After executing shundrecover command, the following message occurs:

shn30332u: There is not enough space in the data file folder. [shund1]

Error Confirmation

The status of the director is "RECOVER". In addition, the status of the director data file is "IOERROR". Therefore, searches and updates of the data through the application cannot be executed.

shundstate -s shund1
Shunsaku shundstate 2006/09/01 13:00:00

Time     State                DataFileStatus       DataSize             Fragments(%)
         LastBackedUp         OperationLogStatus   OperationLogSize
         Records              ReadRecords          ReadTime(sec)        ReadSize

13:00:00 RECOVER              IOERROR             9.67MB               66
         2006/08/31 22:00:00  NORMAL               20.8MB
         10000                10000                0.829                9.78MB

Cause

The director data file directory has no available space during the shundrecover command and the recovery process has been terminated.

Corrective Action

Create some more space in the director data file directory or replace it with a larger capacity disk. Then restore the data again.

Stop the Shunsaku system:

Stop the system forcibly because the status of the director is "RECOVER".

shunsysstop -n shunsaku -e
shun: INFO: shn30802i: Forced stop of Shunsaku system has started. [shunsaku]
shun: INFO: shn30582i: Stopping conductor. [shunsaku]
shun: INFO: shn30583i: Stopping sorter. [shunsaku]
shun: INFO: shn30584i: Stopping director. [shunsaku]
shun: INFO: shn30803i: Shunsaku system has been forcibly stopped. [shunsaku]

Create more space in the director data file directory or replace it with a larger capacity disk.

Restart the Shunsaku system

shunsysstart -n shunsaku (Shunsaku system name)
shun: INFO: shn30567i: Activating Shunsaku system. [shunsaku]
shun: INFO: shn30573i: Activating director. [shunsaku]
shun: INFO: shn30572i: Activating sorter. [shunsaku]
shun: INFO: shn30571i: Activating conductor. [shunsaku]

Execute the shundstate command to confirm the status of the director and the data file. Because of the recovery error, the status of the director data file will be "IOERROR".

shundstate -s shund1
Shunsaku shundstate 2006/09/01 13:10:00

Time     State                DataFileStatus       DataSize             Fragments(%)
         LastBackedUp         OperationLogStatus   OperationLogSize
         Records              ReadRecords          ReadTime(sec)        ReadSize

13:10:00 ACTIVE               IOERROR             9.67MB               66
         2006/08/31 22:00:00  NORMAL               20.8MB
         10000                10000                0.829                9.78MB

Execute the shundrecover command to declare the start of the recovery:

shundrecover -s shund1 -b
shun: INFO: shn21001i: Starting process... [shund1]
shun: INFO: shn30327i: Recovery start declaration processing has been completed. [shund1]
shun: INFO: shn21002i: Processing has been completed. [shund1]

Restore the director data file backup data again.

Execute the shundrecover command to recovery the system. The following example is of a message showing the recovery to its latest state:

shundrecover -s shund1 -e
shun: INFO: shn21001i: Starting process... [shund1]
shun: INFO: shn30334i: Recovery end declaration processing is complete. [shund1]
shun: INFO: shn21002i: Processing has been completed. [shund1]

Execute the shundstate command to confirm whether the status of the director is "ACTIVE" and the status of the director data file is "NORMAL":

shundstate -s shund1
Shunsaku shundstate 2006/09/01 13:20:00

Time     State                DataFileStatus       DataSize             Fragments(%)
         LastBackedUp         OperationLogStatus   OperationLogSize
         Records              ReadRecords          ReadTime(sec)        ReadSize

13:20:00 ACTIVE               NORMAL                9.67MB               66
         2006/08/31 22:00:00  NORMAL                20.8MB
         10000                10000                 0.829                9.78MB

Execute the shundbackup command to back up the director data file.
Recommence operations.

An I/O error in the operation log file

Error Event

After executing the shundrecover command, the following message is output:

shn30335u: An I/O error has occurred with an operation log file.errno= d*

[shund1]

d*: Error number for the system call

An I/O error has occurred in the disk where the operation log file is located.

Error Confirmation

The status of the director is "RECOVER". In addition, the status of the director data file is "IOERROR". Therefore, data cannot be searched or updated through the application.

shundstate -s shund1
Shunsaku shundstate 2006/09/01 13:00:00

Time     State                DataFileStatus       DataSize             Fragments(%)
         LastBackedUp         OperationLogStatus   OperationLogSize
         Records              ReadRecords          ReadTime(sec)        ReadSize

13:00:00 RECOVER              NORMAL                9.67MB               66
         2006/08/31 22:00:00  IOERROR             20.8MB
         10000                10000                0.829                9.78MB

Corrective Action

First, recover the operation log file. Then recover the director data file. However, only data from the backup point can be restored because the operation log file is corrupt.

Replace the disk where the operation log file is located. Then recover the operation log file.

shundrecover -s shund1 -l
shun: INFO: shn21001i: Starting process... [shund1]
shun: INFO: shn30344i: Operation log files have been recovered. [shund1]
shun: INFO: shn21002i: Processing has been completed. [shund1]

If the operating system must be restarted during disk replacement, stop the Shunsaku system and restart it after replacing the disk.

Execute the shundstate command to confirm the status of the director and the data file.

If the status of the director is "NORMAL", execute the shundrecover command to declare the start of the recovery:

shundrecover -s shund1 -b
shun: INFO: shn21001i: Starting process... [shund1]
shun: INFO: shn30327i: Recovery start declaration processing has been completed. [shund1]
shun: INFO: shn21002i: Processing has been completed. [shund1]

Restore the director data file backup data again.

Execute the shundrecover command to restore the data to the recovery point:

shundrecover -s shund1 -e -p
shun: INFO: shn21001i: Starting process... [shund1]
shun: INFO: shn30334i: Recovery end declaration processing is complete. [shund1]
shun: INFO: shn21002i: Processing has been completed. [shund1]

Execute the shundstate command to confirm whether the status of the director is "ACTIVE" and the status of the director data file is "NORMAL":

shundstate -s shund1
Shunsaku shundstate 2006/09/01 13:10:00

Time     State                DataFileStatus       DataSize             Fragments(%)
         LastBackedUp         OperationLogStatus   OperationLogSize
         Records              ReadRecords          ReadTime(sec)        ReadSize

13:10:00 ACTIVE               NORMAL                9.67MB               66
         2006/08/31 22:00:00  NORMAL                20.8MB
         10000                10000                 0.829                9.78MB

Execute the shundbackup command to back up the director data file.
Recommence operations.

Not the latest backup data

Error Event

After the shundrecover command is executed, the following recovery error is output:

shn30338u: There is an error with the backup data that has been restored. [shund1]

The data cannot be recovered because the restored director data file is not backup data collected at the latest backup point.

Cause

The restored backup data in the director data file is not the data that is collected at the latest backup point displayed at the "LastBackedUp" by the shundstate command.

Consistency between the backup data and the update log data cannot be obtained in the recovery process (either restoring backup data to its latest state or to the specified point), which applies the update log data accumulated in the operation log file. Therefore, this error message is output and the recovery process is in error.

[Example of outputting the error message shn30338u]

Corrective Action

Restore the latest collected backup data. Then declare the recovery end by executing the shundrecover command with the -e option.

If the status of the director displayed by the shundstate command is "ACTIVE", declare the start of recovery. This is not necessary if the status of the director is "RECOVER".

shundrecover -s shund1 -b
shun: INFO: shn21001i: Starting process... [shund1]
shun: INFO: shn30327i: Recovery start declaration processing has been completed. [shund1]
shun: INFO: shn21002i: Processing has been completed. [shund1]

Restore the latest collected backup data.
Execute the shundrecover command to declare the end of recovery.

The following example is of a message that shows the recovery to its latest state:

shundrecover -s shund1 -e
shun: INFO: shn21001i: Starting process... [shund1]
shun: INFO: shn30334i: Recovery end declaration processing is complete. [shund1]
shun: INFO: shn21002i: Processing has been completed. [shund1]

Execute the shundstate command to confirm whether the status of the director is "ACTIVE":

shundstate -s shund1
Shunsaku shundstate 2006/09/01 13:10:00

Time     State                DataFileStatus       DataSize             Fragments(%)
         LastBackedUp         OperationLogStatus   OperationLogSize
         Records              ReadRecords          ReadTime(sec)        ReadSize

13:10:00 ACTIVE               NORMAL                9.67MB               66
         2006/08/31 22:00:00  NORMAL                20.8MB
         10000                10000                 0.829                9.78MB

Execute the shundbackup command to back up the director data file.
Recommence operations.

Discontinuous update log data error

Error Event

After executing the shundrecover command to restore the backup data to its latest state or to the specified point, a recovery error may occur and the following message is output:

shn30341w: Recovery processing has been completed. Recovery point=s* Cause=t* [u*]

s*: Time.

t*: Reason for recovery finishing at the time shown.

u*: Director identifier.

Recovery processing could only be executed up to the time shown, due to the reason indicated by "Cause".

Error Confirmation

An error is considered as a discontinuous update log data error if the error message described in Error Event above is output.

The recovery process will restore the data to the time that the message is displayed. Because the status of the director is "ACTIVE", data from the range that has been restored can be searched and updated.

Cause

The shundrecover command was executed to restore the backup data to its latest state or to/from any point but it detected that the update log was in a discontinuous state. Because of this, the recovery process was terminated and an error message was output.

The update log data in the operation log file cannot be restored (either to the latest state or the specified point). Refer to the table below for the specific reason.

In this case, the recovery process is terminated and a message appears when the data has been restored to a recoverable point.

The following table explains why the update log data is not applicable:

Cause	Reason	Recovery Point
shundimport -n	The shundimport command with the -n option (do not collect update log data) was executed.	Data can be restored to the state before the shundimport command was executed.
shundrecover -p	The shundrecover command with the -p option (specifying the time or the backup point) was executed to recover.	Data can be restored to the state before executing recovery to the specified time or back up point.
operation log is discontinuous	The update log is discontinuous because the operation log file was recovered or an operation log file that cannot be collected was updated.	Data can be restored to the state before the update log data became discontinuous.

Corrective Action

If this message is output during the recovery process, the state after the message is displayed will not be able to be recovered.

To avoid this problem, back up the director data file before doing any of the following:

Executing the shundimport command with the -n option to import or delete XML documents.
Executing the shundrecover command with the -e option or the -p option to recover the system to the back up point or the specified back up point.
Executing the shundrecover command with the -i option to recover the operation log file.
Changing the "OperationLogFolder" parameter in the director environment file.

A character code error

Error Event

After executing the shundrecover command, the following message is output:

shn30355u: The data which is not well-formed XML document exists. [shund1]

There is an XML document that is not well-formed or there is an error in the character code.

Cause

The following character codes are different:

The character code specified by the CharacterCode parameter in the system environment file.
The character code of the backup data in the director data file.

Corrective Action

Specify that the same character code for the CharacterCode parameter in the system environment file and the backup data in the director data file. Then restore the backup data again.

Stop the Shunsaku system

shunsysstop -n shunsaku (Shunsaku system name)
shn30579i:Stopping Shunsaku system. [shunsaku]
shn30582i:Stopping conductor. [shunsaku]
shn30583i:Stopping sorter. [shunsaku]
shn30584i:Stopping director. [shunsaku]
shn30580i:Shunsaku system has been stopped. [shunsaku]

Specify the same character code for the CharacterCode parameter of the system environment file as for the backup data in the director data file. Then distribute to all servers.

Start the Shunsaku system.

shunsystart -n shunsaku (Shunsaku system name)
shn30567i:Activating Shunsaku system. [shunsaku]
shn30573i:Activating director. [shunsaku]
shn30572i:Activating sorter. [shunsaku]
shn30571i:Activating conductor. [shunsaku]

Declare the start to recovery.

shundrecover -s shund1 -b
shun: INFO: shn21001i: Starting process... [shund1]
shun: INFO: shn30327i: Recovery start declaration processing has been completed. [shund1]
shun: INFO: shn21002i: Processing has been completed. [shund1]

Restore the director data backup data file again.
Execute the shundrecover command to declare the end of recovery.

The following example is of a message showing the recovery to its latest state:

shundrecover -s shund1 -e
shun: INFO: shn21001i: Starting process... [shund1]
shun: INFO: shn30334i: Recovery end declaration processing is complete. [shund1]
shun: INFO: shn21002i: Processing has been completed. [shund1]

Execute the shundstate command to confirm whether the status of the director is "ACTIVE":

shundstate -s shund1
Shunsaku shundstate 2006/09/01 13:10:00

Time     State                DataFileStatus       DataSize             Fragments(%)
         LastBackedUp         OperationLogStatus   OperationLogSize
         Records              ReadRecords          ReadTime(sec)        ReadSize

13:10:00 ACTIVE               NORMAL                9.67MB               66
         2006/08/31 22:00:00  NORMAL                20.8MB
         10000                10000                 0.829                9.78MB

Execute the shundbackup command to back up the director data file.
Recommence operations.

A searcher error

Error Event

After executing shundrecover command, the following message is output:

shn30364u: The director is not ready to receive requests. [shund1]

Cause

The recovery process cannot be continued because a searcher error was detected during the declaration to start recovery after the shundrecover command was executed with the -b option and the declaration to stop recovery was specified by the -e option.

In the recovery process, the information of the currently used searcher will be written into the director data file. If the searcher error was detected during the recovery process, the director data file will suspend the recovery process. Therefore, it is necessary to restore the backup data again.

Corrective Action

After recovering the searcher, retry the recovery process from the restoring backup data with the following process:

Recover the error searcher then start it.

If the status of the director displayed by the shundstate command is "ACTIVE", declare the start of recovery. This is not necessary if the status of the director is "RECOVER".

shundrecover -s shund1 -b
shun: INFO: shn21001i: Starting process... [shund1]
shun: INFO: shn30327i: Recovery start declaration processing has been completed. [shund1]
shun: INFO: shn21002i: Processing has been completed. [shund1]

Restore the director data file backup data again.
Execute the shundrecover command to declare the end of recovery.

The following example is of a message showing the recovery to its latest state:

shundrecover -s shund1 -e
shun: INFO: shn21001i: Starting process... [shund1]
shun: INFO: shn30334i: Recovery end declaration processing is complete. [shund1]
shun: INFO: shn21002i: Processing has been completed. [shund1]

Execute the shundstate command to confirm whether the status of the director is "ACTIVE":

shundstate -s shund1
Shunsaku shundstate 2006/09/01 13:10:00

Time     State                DataFileStatus       DataSize             Fragments(%)
         LastBackedUp         OperationLogStatus   OperationLogSize
         Records              ReadRecords          ReadTime(sec)        ReadSize

13:10:00 ACTIVE               NORMAL                9.67MB               66
         2006/08/31 22:00:00  NORMAL                20.8MB
         10000                10000                 0.829                9.78MB

Execute the shundbackup command to back up the director data file.
Recommence operations.

Contents Index