When a processing error occurs or is detected inside a coordination adapter, a processing error email is sent.
Investigate the cause of the error using the following procedure:
Confirm the subscription ID of the processing error email.
The subscription ID is included in the title (subject) of the processing error email.
View access and error logs.
In the access and error logs, search for the subscription ID confirmed in step 1.
Investigate the cause of the error and take corrective action.
From the logs searched in step 2, investigate the cause of the error and take appropriate corrective action.
Email Subject
<Header> <Subscription ID> <Error Message>
In the header, a value corresponding to the operation type is configured.
In Subscription ID, one of the following value formats is configured.
Virtual Machine Operation Type | Format |
|---|---|
Virtual machines | <Service ID> <Cloud Type> <Date and Time> |
Creation of Snapshots | <Service ID> <Cloud Type> <Date and Time> |
Restoration of Snapshots, Deletion of Snapshots | <Service ID> <Cloud Type> <Snapshot ID> <Date and Time> |
Key Pairs | <Platform ID> <Cloud Type> <Date and Time> |
In the individual components of the format for the Subscription ID, the following values are configured:
Service ID: The service ID entered in the [Add Service] window
Platform ID: The platform ID entered in the [Add Platform] window
Cloud Type: A cloud type is displayed in the format, ess.xxxxxx.
In xxxxxx, aws, k5, ror, azure, or vmware is configured.
Date and Time: The date and time when the error occurred is displayed in the format, yyyymmddhhmmssSSS.
For the error message, an overview of the error is configured.
Email Body
The body of the email contains the error message and detailed content. The email is sent using the following format:
Subject: Provisioning process for subscription 'fjhvmu1aw ess.aws 20160229172725787'aborted -------------------------------------- Dear user, The provisioning process for subscription 'fjhvmu1aw ess.aws 20160229172725787' for customer '!mgr' has been aborted (instance: 'aws-xxxxxxxx-yyyy-zzzzzzzzz-aaaaaaaaaaaa'), Reported problem: AWS returned the following error: Unable to execute HTTP request: xxx.xx-xxxxxxxxx-1.xxxxxx.com
Information
The URL of the Browser Interface for Management of the Coordination Adapter Platform is contained in the email body.
For details on the Browser Interface for Management of the Coordination Adapter Platform, refer to "Appendix B Operation of the Browser Interface for Coordination Adapter Base Management".
Access Logs
The access logs of all coordination adapters are output to the following file:
%FSCSM_HOME%\SWCTMG\glassfish3\glassfish\domains\app-domain\logs\app-core.log
Information
When the size of an access log exceeds 10 MB, old information is saved as a file with a serial number appended to the file name.
app-core.log.1
A log in the following format is output:
[02/08 10:28:28] [http-thread-pool-6081(5)] INFO Create instance for organization for subscription ServiceName002_userId_20160208102820.001.
Error Logs
Detailed information of the content that is output at the log levels "WARN" and "ERROR" in an access log is output to another file as an error log.
Error logs of coordination adapters are output to the following location.
%FSCSM_HOME%\SWCTMG\glassfish3\glassfish\domains\app-domain\logs
Individual log files are created for each coordination adapter.
AWS
app-aws.log
Azure
app-azure.log
ROR
app-ror.log
K5
app-k5.log
VMware
app-vmware.log
Information
When the size of an error log exceeds 10 MB, old information is saved as a file with a serial number appended to the file name.
app-ror.log.1
A log in the following format is output:
[12/08 16:59:50] [__ejb-thread-pool15] ERROR Error while checking instance status org.oscm.app.v1_0.exceptions.AuthenticationException: User does not belong to the correct organization. at rg.oscm.app.v1_0.service.APPAuthenticationServiceBean.authenticateUser(APPAuthenticationServiceBean.java: 196) at org.oscm.app.v1_0.service.APPAuthenticationServiceBean.getAuthenticatedTMForController(APPAuthentication ServiceBean.java:116) at org.oscm.app.v1_0.service.__EJB31_Generated__APPAuthenticationServiceBean__Intf____Bean__.getAuthenticated TMForController(Unknown Source) at org.oscm.app.v1_0.service.APPlatformServiceBean.authenticate(APPlatformServiceBean.java:149) at org.oscm.app.v1_0.intf.__APPlatformService_Remote_DynamicStub.authenticate(org/oscm/app/v1_0/intf/__APPlat formService_Remote_DynamicStub.java) at org.oscm.app.v1_0.intf._APPlatformService_Wrapper.authenticate(org/oscm/app/v1_0/intf/_APPlatformService_ Wrapper.java)
The following are possible causes of the error:
Issues in the management server settings
The required parameters are not specified in the cloud vendor information.
Inappropriate values of parameters are specified in the cloud vendor information.
Problems in the communication between the coordination adapter and cloud
During setup of the coordination adapter, the information for connecting to the coordinating cloud was set incorrectly.
During setup of the coordination adapter, the information for connecting to an external proxy was set incorrectly.
Error occurring on the cloud side
A resource (server name, etc.) that must be unique is already being used.
The number of resources that can be created on the cloud has reached its upper limit.
For AWS
When a proxy server for external connections is not correctly configured to connect to a cloud
Unable to execute HTTP request: <HTTP_server_name>
Confirm the following:
The connection information of the proxy server for external connections is configured correctly.
The proxy server is up and running.
For Azure
When no subscription files are added
An error has occurred. detail=Subscription file is not found. path=<Folder Name>
It is necessary to add a subscription file for connecting to Azure.
For the procedure for adding subscription files, refer to "Configuring Azure Connection Information (Only when Performing Azure Coordination)" in the "FUJITSU Software Cloud Services Management Installation Guide".
When a proxy server for external connections is not correctly configured to connect to a cloud
An error has occurred. detail=<HTTP_server_name>
Confirm the following:
The connection information of the proxy server for external connections is configured correctly.
The proxy server is up and running.
For ROR
When the parameters of a coordination adapter for a cloud are not correctly configured
Authorization failed for the user[admin_].
Check the parameters that were specified using the setup command of the coordination adapter.
When configurations to connect to a cloud have not been performed correctly
Command failed: ListLPlatform Message: Stream closed
Confirm the following:
The connection information of the proxy server for external connections is configured correctly.
The proxy server is up and running.
The host name of and port number for the cloud specified using the setup command of the coordination adapter are correctly configured.
For K5
When the parameters of a coordination adapter for a cloud are not correctly configured
The request you have made requires authentication.
Check the parameters that were specified using the setup command of the coordination adapter.
When a proxy server for external connections is not correctly configured to connect to a cloud
Command failed: POST https://<K5_server_name>/v3/auth/tokens Message: <HTTP_server_name>
Confirm the following:
The connection information of the proxy server for external connections is configured correctly.
The proxy server is up and running.
The user ID and password of the cloud specified using the setup command of the coordination adapter are correctly configured.
For VMware
When the parameters of a coordination adapter for a cloud are not correctly configured
VMwareClient connect Failed to login. URL: <URL_to_be_connected> UserId: <User ID>
Check the parameters that were specified using the setup command of the coordination adapter.
When a proxy server for external connections is not correctly configured to connect to a cloud
Unable to tunnel through proxy. Proxy returns "HTTP/1.1 407 Proxy Authentication Required"
Confirm the following:
The connection information of the proxy server for external connections is configured correctly.
The proxy server is up and running.
The user ID and password of the cloud specified using the setup command of the coordination adapter are correctly configured.
Common
When a coordination adapter for the cloud has not been set up
The controller configuration is missing the following parameter(s): <Parameter Name>
Confirm that the coordination adapter and certificates have been correctly set up.
When email transmission fails because connection to the SMTP server cannot be made
Failure during error mail notification for service instance '<Image ID>' with message 'Mail could not be sent. [Cause: Unknown SMTP host: <SMTP Server Name>]'
Confirm the following:
The SMTP server settings specified using the setup command of the coordination adapter platform (APP) and the management server are correct.
The SMTP server is running.
When login to a virtual machine cannot be performed
The correct password may have not been sent because the "OS_TYPE" of the cloud vendor definition file and the OS type of the image do not match.
In this case, no logs are output.
When the OS types do not match, recreate the virtual machine.
For K5
When the maximum number of virtual machines available for use in a K5 project is exceeded
Max servers have been deployed to all of k5 projects.
Delete unnecessary virtual machines. Alternatively, increase the number of virtual machines available for use on the cloud and then modify the value of the "PROJECT_MAX_VM" parameter of the K5 coordination adapter.
Common
When the required parameters are not specified in the cloud vendor definition file
The '<Parameter Name>' is mandatory.
Check the configured content in the cloud vendor definition file. For details on required parameters, refer to "2.10.2 Cloud Vendor Information".
When an invalid value is specified for a parameter in the cloud vendor definition file
'<Parameter Name> <Parameter Value>' does not exist
A value that does not exist on the cloud may be specified in the cloud vendor definition file. Check the information on the cloud and correct the parameter in the cloud vendor definition file.
For AWS
When resources on a cloud are insufficient
The account already has the maximum number of pipelines allowed: account=<accountId>, maximum number of pipelines=<maximum>
This problem occurs when there are insufficient resources on a cloud when creating a virtual machine.
Check the amount of free space of the resources on the cloud. When there are insufficient resources, delete unnecessary resources (such as virtual machines that have already been created) and create a virtual machine again.
When the number of additional block devices that can be specified on a cloud is exceeded
The number of block device is larger than the number of additional block device name.
Check the range of the parameter specifiable on the cloud and then specify the parameter value.
In addition, when an error occurs during the reconfiguration of virtual machines, additional block devices that are not attached to virtual machines may remain on the cloud. Delete the disks that are not attached using the management console of the cloud and then perform reconfiguration again.
When the creation of a snapshot is attempted while a virtual machine is running
EC2 instance is not in the stopped state.
Before performing the reconfiguration of virtual machines and the creation and restoration of snapshots, virtual machines must be stopped in advance.
Stop the virtual machine and then create the snapshot.
For Azure
When the number of additional block devices that can be specified on a cloud is exceeded
Too many data disks specified for virtual machine '<Virtual Machine Name>'. The maximum number of data disks currently permitted is '<Upper Limit>'. The current number of data disks is '<Parameter Value>'. The operation is attempting to add 1 additional data disks.
Check the range of the parameter specifiable on the cloud and then specify the parameter value.
When the maximum size of additional block devices that can be specified on a cloud is exceeded
The specified disk size value of '<Parameter Value>' GB is invalid. Disk size must be between '<Specified Range>'.
Check the range of the parameter specifiable on the cloud and then specify the parameter value.
In addition, when an error occurs during the reconfiguration of virtual machines, additional block devices that are not attached to virtual machines may remain on the cloud. Delete the disks that are not attached using the management console of the cloud and then perform reconfiguration again.
For ROR
When resources on a cloud are insufficient
Selectable VM host not found. (not enough CPU or memory available)
This problem occurs when there are insufficient resources on a cloud when creating a virtual machine.
Check the amount of free space of the resources on the cloud. When there are insufficient resources, delete unnecessary resources (such as virtual machines that have already been created) and create a virtual machine again.
When the number of additional block devices that can be specified on a cloud is exceeded
Command failed: CreateDisk Response Code: <HTTP Status Code> Status: VALIDATION_ERROR Message: '<Disk Name>' The vdisks' number can't exceed the max number of disks according to the image Id assigned.
Check the range of the parameter specifiable on the cloud and then specify the parameter value.
In addition, when an error occurs during the reconfiguration of virtual machines, additional block devices that are not attached to virtual machines may remain on the cloud. Delete the disks that are not attached using the management console of the cloud and then perform reconfiguration again.
When two or more virtual machines are defined in an L-Platform template
Specified virtual system template ID '<Template ID>' has '<Number of Virtual Machines>' servers. It must have only one.
Define only one virtual machine in an L-Platform template.
For K5
When resources on a cloud are insufficient
Virtual server '<Virtual Machine Name>' is in error state.
This problem occurs when there are insufficient resources on a cloud when creating a virtual machine.
Check the amount of free space of the resources on the cloud. When there are insufficient resources, delete unnecessary resources (such as virtual machines that have already been created) and create a virtual machine again.
When the number of additional block devices that can be specified on a cloud is exceeded
communication error. (jp-east-1b :status=<Error Code> :{"overLimit": {"message": "VolumeLimitExceeded: Maximum number of volumes allowed (50) exceeded", "code": <Error Code>}})Check the range of the parameter specifiable on the cloud and then specify the parameter value.
In addition, when an error occurs during the reconfiguration of virtual machines, additional block devices that are not attached to virtual machines may remain on the cloud. Delete the disks that are not attached using the management console of the cloud and then perform reconfiguration again.
When the creation of a snapshot is attempted while a virtual machine is running
In server state ACTIVE, cannot create snapshot.
Before performing the reconfiguration of virtual machines and the creation and restoration of snapshots, virtual machines must be stopped in advance.
Stop the virtual machine and then create the snapshot.
When there are no K5 projects on a cloud
The request you have made requires authentication.
Create a virtual machine specifying a valid K5 project. Or create a K5 project on the cloud.
For VMware
When resources on a cloud are insufficient
Actions inspectTaskResult File <unspecified filename> is larger than the maximum size supported by datastore '<unspecified datastore>'
This problem occurs when there are insufficient resources on a cloud when creating a virtual machine.
Check the amount of free space of the resources on the cloud. When there are insufficient resources, delete unnecessary resources (such as virtual machines that have already been created) and create a virtual machine again.
When the number of additional block devices that can be specified on a cloud is exceeded
A specified parameter was not correct: unitNumber
Check the range of the parameter specifiable on the cloud and then specify the parameter value.
In addition, when an error occurs during the reconfiguration of virtual machines, additional block devices that are not attached to virtual machines may remain on the cloud. Delete the disks that are not attached using the management console of the cloud and then perform reconfiguration again.
When no resource pools have been created on a cloud
Resourcepool not found.
Check whether resource pools have been created on the cloud. If no resource pools have been created, create one.
Common
When creation of a virtual machine does not complete
This refers to a case where the status of a virtual machine does not change from "deploying". In this case, confirm the following:
Check the log for the coordination adapter. If an error message is displayed, take corrective action according to the error content (the log for the management server is not output).
Check the status of the virtual machine from the portal of the cloud. If the process for creation of the virtual machine does not complete, contact the support center or take corrective action for the corresponding cloud.
When consistency of management information is not ensured between Cloud Services Management and a cloud, perform recovery referring to "6.2 Ensuring Consistency of Management Information".
Note
The content of the error messages for AWS, Azure, and K5 described here may change when the functions of each cloud service are updated.
Once the problem is identified, take the necessary actions to solve it and then confirm that the error no longer occurs.
See
For the parameters set in the cloud vendor information, refer to "2.3.1 Cloud Preparations". For configuring cloud vendor information, refer to "2.10.2 Cloud Vendor Information".
For details regarding the setup of coordination adapters, refer to "Setup of Coordination Adapters" in the "FUJITSU Software Cloud Services Management Installation Guide".
Use the management consoles of each cloud services to confirm their respective configurations.
For details on the Browser Interface for Management of the Coordination Adapter Platform, refer to "Appendix B Operation of the Browser Interface for Coordination Adapter Base Management".
Collect troubleshooting data and contact the Fujitsu support staff if the problem cannot be identified or the trouble cannot be resolved by the above actions.
See
For details regarding the collection of troubleshooting data, refer to "4.3 Data Collection Tool".
Common
When emails from a coordination adapter are not received, and the platform providers receive a completion notification (error) email
In the Cloud Management Portal, if a virtual machine is still in the status from before an operation was performed, a platform provider should perform the relevant operation again.