Autologon windows in Application Director

I had a customer who was trying to deploy a blueprint from Application Director and need autologon feature. The Endpoint is vCloud Director

AppD –> vCloud Director –> VM

When we deployed the VM directly from vCD (Request –> vCD –> VM) , autologon worked but same didn’t work from AppD (Request –> AppD –> vCD –>VM).

In the AppD documentation, I noticed that when we deploy from AppD, vCD Guest customization will be over-ridden.

So simple idea is to add a windows powershell script to add in AppD (AppD console –> Powershell script –>paste), below script flawlessly


$VMname= hostname

set-ItemProperty -Path 'HKLM:SOFTWAREMicrosoftWindows NTCurrentVersionWinlogon' -Name AutoAdminLogon -Value 1
set-ItemProperty -Path 'HKLM:SOFTWAREMicrosoftWindows NTCurrentVersionWinlogon' -Name DefaultUserName -Value Administrator
set-ItemProperty -Path 'HKLM:SOFTWAREMicrosoftWindows NTCurrentVersionWinlogon' -Name DefaultPassword -Value SWENGn2j4g5
set-ItemProperty -Path 'HKLM:SOFTWAREMicrosoftWindows NTCurrentVersionWinlogon' -Name DefaultDomainName -Value $VMname
set-ItemProperty -Path 'HKLM:SOFTWAREMicrosoftWindows NTCurrentVersionWinlogon' -Name ForceUnlockLogon -Value 1
set-ItemProperty -Path 'HKLM:SOFTWAREMicrosoftWindows NTCurrentVersionWinlogon' -Name AutoLogonCount -Value 100

Log Locations of all VMware Products

In this post I tried consolidating all the VMware product’s log location for handy use.

 

vCenter

The vCenter Server logs are placed in a different directory on disk depending on vCenter Server version and the deployed platform:
vCenter Server 5.x and earlier versions on Windows XP, 2000, 2003: %ALLUSERSPROFILE%Application DataVMwareVMware VirtualCenterLogs
vCenter Server 5.x and earlier versions on Windows Vista, 7, 2008: C:ProgramDataVMwareVMware VirtualCenterLogs
vCenter Server 5.x Linux Virtual Appliance: /var/log/vmware/vpx/
vCenter Server 5.x Linux Virtual Appliance UI: /var/log/vmware/vami

Note: If the service is running under a specific user, the logs may be located in the profile directory of that user instead of %ALLUSERSPROFILE%.

Usage Log File Name
The main vCenter Server logs, consisting of all vSphere Client and WebServices connections, internal tasks and events, and communication with the vCenter Server Agent (vpxa) on managed ESX/ESXi hosts. vpxd.log
Profiled metrics for operations performed in vCenter Server. Used by the VPX Operational Dashboard (VOD) accessible at https://VCHostnameOrIPAddress/vod/index.html. vpxd-profiler.logprofiler.log
Non-fatal information logged about the vpxd process. vpxd-alert.log
Common Information Model monitoring information, including communication between vCenter Server and managed hosts’ CIM interface cim-diag.log and vws.log
Actions proposed and taken by VMware Distributed Resource Scheduler (DRS), grouped by the DRS-enabled cluster managed by vCenter Server. These logs are compressed. drmdump:
Health reports for the Licensing Services extension, connectivity logs to vCenter Server. ls.log
Dump of string used during the installation of vCenter Server with hashed information for DNS, username and output for JDBC creation. vimtool.log
Provides information about the historical performance data collection from the ESXi/ESX hosts stats.log
Health reports for the Storage Monitoring Service extension, connectivity logs to vCenter Server, the vCenter Server database and the xDB for vCenter Inventory Service. sms.log
Health reports for the ESX Agent Monitor extension, connectivity logs to vCenter Server. eam.log
Connectivity information and status of the VMware Webmanagement Services. catalina.<date>.log and localhost.<date>.log
Health status of the VMwareVCMSDS service and individual ADAM database objects, internal tasks and events, and replication logs between linked-mode vCenter Servers. jointool.log

 

vCenter 6.0

The VMware vCenter Server 6.0 logs are located in the %ALLUSERSPROFILE%VMWarevCenterServerlogs folder.
The VMware vCenter Server Appliance 6.0 logs are located in the /var/log/vmware/ folder.

vCenter Server logs are grouped by component and purpose in these sub-directories:

vCenter Server
vCenter Server Appliance
Description
vmware-vpxvpxd.log
vpxd/vpxd.log
The main vCenter Serverlog
vmware-vpxvpxd-profiler.log
vpxd/vpxd-profiler.log
Profile metrics for operations performed in vCenter Server
vmware-vpxvpxd-alert.log
vpxd/vpxd-alert.log
Non-fatal information logged about the vpxd process
perfchartsstats.log
perfcharts/stats.log
VMware Performance Charts
eameam.log
eam/eam.log
VMware ESX Agent Manager
invsvc
invsvc
VMware Inventory Service
netdump
netdumper
VMware vSphere ESXi Dump Collector
vapi vapi VMware vAPI Endpoint
vmdird
vmdird
VMware Directory Service daemon
vmsyslogcollector
syslog
vSphere Syslog Collector
vmware-spssps.log
vmware-sps/sps.log
VMware vSphere Profile-Driven Storage Service
vpostgres
vpostgres
vFabric Postgres database service
vsphere-client
vsphere-client
VMware vSphere Web Client
vws
vws
VMware System and Hardware Health Manager
workflow
workflow
VMware vCenter Workflow Manger
SSO SSO VMware Single Sign-On

 

Back to top

vSphere Profile-Driven Storage log files

The vSphere Profile-Driven Storage logs are placed in a different directory on disk depending on the vCenter Server version and the deployed platform:

  • vCenter Server 5.x and earlier versions on Windows XP, 2000, 2003: %ALLUSERSPROFILE%Application DataVMwareInfrastructureProfile-Driven StorageLogs
  • vCenter Server 5.x and earlier versions on Windows Vista, 7, 2008: C:ProgramDataVMwareInfrastructureProfile-Driven StorageLogs
  • vCenter Server 5.x Linux Virtual Appliance: /var/log/vmware/vpx/spsNote: If the service is running under a specific user, the logs may be located in the profile directory of that user instead of%ALLUSERSPROFILE%.
Usage Log
The main Profile-Driven Storage logs, consisting of all vCenter Server and Management Webservices connections, internal tasks and events, and information about the storage profile integrity. sps.log
This file consists of information about the installation of Profile-Driven Storage including computer name, operating system revision, the date of installation and the number of revisions that have been installed or upgraded on the system. vim-sps-install.log
This file provides information about the state of the Java runtime environment. wrapper.log

Back to top

vSphere Web Client service logs

For vSphere 5.0, all the logs for the vSphere Web Client service are located at:

  • Windows: C:Program FilesVMwareInfrastructurevSphere Web ClientDMServerserviceability
  • Linux: /usr/lib/vmware-vsphere-client/server/serviceability/

For vSphere 5.1, all the logs for the vSphere Web Client service are located at:

  • Windows: C:ProgramDataVMwarevSphere Web Clientserviceability
  • Linux: /var/log/vmware/vsphere-client/serviceability/

For vSphere 5.5, all the logs for the vSphere Web Client service are located at:

  • Windows: C:ProgramDataVMwarevSphere Web Clientserviceability
  • Linux: /var/log/vmware/vsphere-client/

 

Usage Log
The main vSphere Web Client log, consisting of all communication between underlying Flex endpoint connections, internal tasks and events logsvsphere_client_virgo.log
Individual HTTP requests from the Web browser logsaccesslocalhost_access_log.<date>.txt
 Information about plug-in bundle communication logsvirgo-serverlog.log
Time stamps of startup and shutdown of the vSphere Web Client service eventlogs/eventlog.log

Back to top

vCenter Single Sign-On

The SSO logs are placed in a different directory on disk depending on the vSphere version and the deployed platform:

 

vSphere 5.1 using vCenter Single Sign-On 5.1:

vCenter Server 5.1.x on Windows Server 2003 – C:Program FilesVMwareInfrastructureSSOServerlogs

vCenter Server 5.1.x on Windows Server 2008/2012 – C:Program FilesVMwareInfrastructureSSOServerlogs

vCenter Server Virtual Appliance 5.1.x – /var/log/vmware/sso/

vSphere 5.5 using vCenter Single Sign-On 5.5:

vCenter Server 5.5.x on 2003 – %ALLUSERSPROFILE%Application DataCISlogs

vCenter Server 5.5.x and 2008, 2012 – C:ProgramDataVMwareCISlogs

vCenter Server Virtual Appliance 5.5.x – /var/log/vmware/sso/

 

Note: If the service is running under a specific user, the logs may be located in the profile directory of that user instead of %ALLUSERSPROFILE%.
Back to top

 SSO 5.1

 

Usage Log
SSO installer log Vminst.log
SSO Lookup Service log vm_vim_ls.log
MSI installer for SSO installation log – verbose vim-sso-msi.log
SSO database operations vim-sso-javalib.log
Standard Tomcat logs catalina.date.log
RSA’s Castle logs imsTrace.log and imsSystem.log
SSO admin serve ssoAdminServer.log
 Lookup server log lookupServer.log
Replication replicationTransport.log

Back to top

 SSO 5.5

Usage Log
SSO installer log Vminst.log
MSI installer for SSO installation log – verbose vim-sso-msi.log
 SSO Service log vim_ssoreg.log
 Endpoint information about each of the SSO Solution Users and identity sources extracted from previous vCenter Single Sign On 5.1.0 instance exported_sso.properties
MSI installer verbose log for OpenSSL installation vim-openssl-msi.log
 MSI installer verbose log for Python installation vim-python-msi.log
MSI installer verbose log for MIT Kerberos installation vim-kfw-msi.log
 Promotion and demotion operation information for the SSO instance when joined or removed from a linked configurations vmdirdvdcpromo.log
 VMware Directory Service setup post-installation log containing information about the localhost vmdirdvdcsetupIdu.log
 Health reports for the VMware Directory Service (VMDir) service and the VMDir database vmdirdvmdir.log
Key Distribution Center (kdc) run-time log, reports ports conflicts preventing the service from starting vmkdcdvmkdcd.log
 VMware Identity Management service run-time logs, time-stamped records of user attempts when accessing SSO for administrative purposes vmware-ssovmware-sts-idmd.log
VMware Identity Management service performance counter logs vmware-ssovmware-sts.ldmd-perf.log
Commons Daemon log once the Identity Management Service has started vmware-ssoVMwareIdentityMgmtService. date.log

Back to top

 vCenter Inventory Service

The vCenter Inventory Service logs are placed in a different directory on disk depending on the vCenter Server version and the deployed platform:

  • vCenter Server 5.x and earlier versions on Windows XP, 2000, 2003: %ALLUSERSPROFILE%Application DataVMwareInfrastructureInventory ServiceLogs
  • vCenter Server 5.x and earlier versions on Windows Vista, 7, 2008:C:ProgramDataVMwareInfrastructureInventory ServiceLogs
  • vCenter Server 5.x Linux Virtual Appliance: /var/log/vmware/vpx/inventoryserviceNote: If the vCenter Inventory Service is running under a specific user, the logs may be located in the profile directory of that user instead of %ALLUSERSPROFILE%.
Usage Log
The main vCenter Inventory Service logs, consisting of all vCenter Server and Single Sign-On connections, internal tasks and events, and information about the xDB. ds.log
This file contains information about the installation of Inventory Service including computer name, operating system revision, the date of installation and the number of revisions that have been installed or upgraded on the system. vim-is-install.log
This file provides information about the status of the Java runtime environment. wrapper.log

Back to top

VMware Converter logs

There are also several ways to diagnose issues by viewing the VMware Converter logs. The logs can contain information that is not apparent from error messages. In newer versions of VMware Converter, you can use the Export Log Data button. Otherwise, logs are typically stored in these directories:

  • Windows NT, 2000, XP, and 2003:
    • C:Documents and SettingsAll UsersApplication DataVMwareVMware Converter EnterpriseLogs
    • C:WINDOWSTempvmware-converter
    • C:WINDOWSTempvmware-temp
  • Windows Vista, 7, and 2008:
    • C:UsersAll UsersApplication DataVMwareVMware Converter EnterpriseLogs
  • Windows 8 and Windows 2012:
  • C:ProgramDataVMwareVMware vCenter Converter Standalonelogs
    • Note: In order to access this location in Windows Vista, 7, or 2008, you may need to go into the folder options and ensure thatShow Hidden Files is enabled and that Hide Protected Operating System Files is disabled.
    • C:WINDOWSTempvmware-converter
    • C:WINDOWSTempvmware-temp
  • Windows NT and 2000:
    • C:WINNTTempvmware-converter
    • C:WINNTTempvmware-temp
  • Linux:
    • $HOME/.vmware/VMware vCenter Converter Standalone/Logs
    • /var/log/vmware-vcenter-converter-standalone

Back to top

vCloud Director

Component Path
Cell Startup log. $VCLOUD_HOME/logs/cell.log
Debugging log messages from the cell $VCLOUD_HOME/logs/vcloud-container-debug.log
Warnings or errors in the cell $VCLOUD_HOME/logs/vcloud-container-info.log
Log of the Watchdog service – which restarts the vCD service if it hangs or stops $VCLOUD_HOME/logs/vmware-vcd-watchdog.log

Back to top

vShield Manager

vSM is a hardened Linux Appliance so use below commands in the vSM to view the logs

Component Commands
vShield Manager Log Show logShow manager logShow manager log followShow manager log last

Back to top

vShield Edge logs

vShield Edge log Show log
Show log follow

Back to top

vCloud Usage Meter

Usage Meter Log /var/log/usgmtr/um.log

Back to top

 

vCloud Connector logs

 

Usage Log
vCloud Connector Server Log /opt/vmware/hcagent/logs/hcs.log
vCloud Connector Node Log /opt/vmware/hcagent/logs/hca.log

Back to top

vCenter Operations

Component VM Path
Admin UI UI /var/log/vmware/admin.log
vSphere UI UI $ALIVE_BASE/user/log/web_std.log
Custom UI UI $ALIVE_BASE/user/log/web.log
CapIQ UI /var/log/vmware/ciq.log
Upgrades UI /var/log/vmware/upgradecoordinator.log
ActiveMQ AL $ALIVE_BASE/user/log/activemq.log
Analytics AL $ALIVE_BASE/user/log/analytics.log
Collector AL $ALIVE_BASE/user/log/collector.log
Adapter AL $ALIVE_BASE/user/log/adapters/<AdapterType>/<AdapterType>_<InstanceID>.log 

 

vRealize Automation Suite 6.x

Back to top

vRealize Automation Virtual Appliance Locations
Description
/var/log/vcac/catalina.out
tc Server Runtime logs, vRealize Automation webapp logs
/var/log/vco/app-server/catalina.out
vRealize Automation’s built-in vRealize Orchectrator logs
/var/log/apache2/access_log
Apache Access logs
/var/log/apache2/error_log
Apache GET/POST Error logs
/var/log/apache2/ssl_request_log
Apache SSL troubleshooting logs
vRealize Automation Infrastructure as a Service Locations
Description
C:Program Files (x86)VMwarevCACAgentsagent_namelogsfile
Plug-in logs example: CPI61, nsx, VC50, VC51Agent, VC51TPM, vc51withTPM, VC55Agent, vc55u, VDIAgent
C:Program Files (x86)VMwarevCACDistributed Execution ManagerDEMORLogsDEMOR_All Distributed Execution Manager logs
C:Program Files (x86)VMwarevCACDistributed Execution ManagerDEMWRLogsDEMWR_All Distributed Execution Worker logs
C:Program Files (x86)VMwarevCACServerLogs Manager Service logs
C:Program Files (x86)VMwarevCACServerConfigToolLogvCACConfiguration-date Repository Configuration logs
C:Program Files (x86)VMwarevCACServerModel Manager DataLogsnothing_today IIS Access logs (usually empty, but can be expected)
C:Program Files (x86)VMwarevCACServerModel Manager WebLogsRepository Repository logs
C:Program Files (x86)VMwarevCACServerWebsiteLogsWeb_Admin_All Web Admin logs
C:inetpublogs IIS logs
Identity Virtual Appliance Locations Description
/var/log/vmware/sso/catalina.out ID VA tc Server Runtime logs
/var/log/vmware/sso/ssoAdminServer.log
SSO Admin Server logsNote: Not applicable to vRealize Automation.
/var/log/vmware/sso/vmware-identity-sts-perf.log STS performance logs
/var/log/vmware/sso/vmware-identity-sts.log STS logs
/var/log/vmware/sso/vmware-sts-idmd-perf.log Identity service performance logs
/var/log/vmware/sso/vmware-sts-idmd.err Identity service error logs
/var/log/vmware/sso/vmware-sts-idmd.log Identity service logs
/var/log/vmware/vmafd/vmafdd.log Identity VA logs
/var/log/vmware/vmdir/vdcsetupldu.log Initial setup logs
/var/log/vmware/vmdir/vmafdvmdirclient.log VMware SSO LDAP initial configuration logs
/var/log/vmware/vmkdc/vmkdcd.log VMware SSO LDAP initial configuration logs
vRealize Application Services Location Description
/home/darwin/tcserver/darwin/logs/catalina.out Application Services tc Server Runtime logs
vMware vRealize Business Standard Description
/usr/local/tcserver/vfabric-tc-server-standard/tcinstance1/logs/catalina.out vRealize Business Advanced and Enterprise tc Server Runtime logs
/usr/local/tcserver/vfabric-tc-server-standard/tcinstance1/logs/auditFile.log REST API requests
/usr/local/tcserver/vfabric-tc-server-standard/tcinstance1/logs/itfm-external-api.log API logs
/usr/local/tcserver/vfabric-tc-server-standard/tcinstance1/logs/itfm-reflib-update.log vRealize Business standard reference library related changes.
/usr/local/tcserver/vfabric-tc-server-standard/tcinstance1/logs/itfm-vc-dc.log Data collector logs
/usr/local/tcserver/vfabric-tc-server-standard/tcinstance1/logs/itfm.log vRealize Business Advanced and Enterprise logs
vCenter Server Appliance (VCSA) 5.5.x Locations Description
/var/log/vmware/vpx/vpxd.log vCenter VPXD logs
/var/log/vmware/vpx/vpxd-alert.log vCenter VPXD alert logs
/var/log/vmware/vpx/vws.log Management Web Service logs
/var/log/vmware/vpx/vmware-vpxd.log vCenter VPXD status change logs
/var/log/vmware/vpx/inventoryservice/ds.log vCenter Inventory Service logs
/var/log/vmware/vsphere-client/logs/vsphere_client_virgo.log vSphere Client logs
/var/log/vmware/vsphere-client/logs/virgo-server/log.log vSphere Client logs
/var/log/vmware/vsphere-client/eventlogs/eventlog.log vSphere Client event logs
vCenter SSO Locations Description
/var/log/vmware/sso/catalina.out SSO tc Server Runtime logs
/var/log/vmware/sso/ssoAdminServer.log SSO Admin Server logs (only in 5.5.x version)
/var/log/vmware/sso/vmware-identity-sts-perf.log STS performance logs
/var/log/vmware/sso/vmware-identity-sts.log STS logs
/var/log/vmware/sso/vmware-sts-idmd-perf.log Identity service performance logs

Back to top

 vCenter Orchestrator

Each VMware vCenter Orchestrator log file serves a different purpose. Review the following to understand what each log file does.

Notes:

  • If you installed Orchestrator standalone, the default installation path is C:Program FilesVMwareOrchestrator.
  • If you installed Orchestrator with the vCenter Server installer, the default installation path is C:Program FilesVMwareInfrastructureOrchestrator.
  • Log file paths listed are relative to the default installation path.
File Default Location Explanation
boot.log app-serverservervmolog In this log you find details on the JBOSS boot, but nothing related to the vCO server. It might be useful in cases where a file from JBOSS is missing or the install is corrupted. This kind of issue is very rare.
boot-state.log app-serverservervmolog In this log you will find the boot state of the vCO server. If the server booted properly it writes an entry together with the vCO version. This file can be useful as a summary of the boot of vCO server. This is included by default in server.log.
scripts-log.log app-serverservervmolog This file contains the thread from all the executions of workflows and actions. It lets you isolate these executions from normal vCO operations. You can find the same information in server.log. If you need to know which workflow was executed you need to look at server.log instead.
server.log app-serverservervmolog
This is the main log of vCO server. It contains the same information as boot-state.log,scripts-log.log in addition to more data. Everything that happens on the vCO server is seen here. When debugging vCO or any application running on vCO (such as LCM), this is the first log you should open.
vmo-configuration.log configurationjettylogs In this log you find the configuration and validation of each component of vCO. This is the jetty service running on the vCO server. The file request.log, in the same directory, might be more useful if you want just a history of actions taken during configuration.
vmware_orchestrator_installer_output.txt
C:tmp
Note: This path is not relative.
This is the log file containing all the information about the installation of vCO. It shows if vCO was silently installed by vCenter or if it was an independent install. It also indicates if the configuration service was started at install or if it was left stopped for security reasons. This file is the only way to detect an issue during an installation.
vso.log apps This is the client log. It is mainly interesting for connection issues with the server and events on the client side.
yyyy-mm-dd.request.log configurationjettylogs This log shows all the elements that were needed to load and display the pages of the configurator. It keeps a history of which actions were taken for the configuration of vCO and when they were done. This might be very useful to identify a change in the behavior of the vCO server after a restart. However, it does not display the value of changed parameters, only that they were changed.
wrapper.log app-serverbin
The wrapper log contains some information that is also found in the server.log. However this is the only place you will be able to know if the VMware vCenter Orchestrator Server service was restarted by the wrapper or by a user.
vCenter_Orchestrator_InstallLog.log On the desktop of the user who installed Orchestrator. This file is created if you abort an installation or the installation fails.

Back to top
Please comment if you need any additional products.

History:

22-June-2015 – Added vCenter 6.0

vCenter Appliance upgrade failed

I was trying to upgrade vCenter Server Appliance 5.0 to vCSA 5.1, there is no inplace upgrade since

 vCSA 5.0 uses IBM DB2 as a DB and

vCSA 5.1 use VMware vPostgres DB ,

so the only available option is migrate the DB

Steps for Upgrade

Keep the vCSA 5.0 Powered on –> Deploy new vCSA 5.1 –> Enable the Trust between both Appliance –> Start Importing.

Detailed steps / videos can be found on below links.

Error that I faced while upgrading was with the iNodes on the new appliance, there were no free iNodes left so that Upgrade failed with error “General system error occurred”

While reviewing the upgrade log /var/log/vmware/vami/upgrade.log there are more than 1 million (10 Lakhs) records on VPX_EVENT_ARG, all the vC event descriptions are stored in this table and it was the cause to fill-up the iNodes .

~ # df -ih 
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sda3   632K  632K   0     100%   /
devtmpfs 0 0 0 - /dev

iNodes are nothing but Index-node to represent the filesystem. for each file it takes each iNode for its ease of accessibility. since it has 1 million records it failed to Index all the files. I was thinking of increasing the iNodes but to increase iNodes it is a requirement to re-create the filesystem, so I approached alternative plan to delete the contents from VPX_EVENT_ARG table.

It was difficult for me to find the commands to truncate / delete the DB. I used below commands to delete the contents from VPX_EVENT_ARG table.

 

  1. Backup and Snapshot the vCenter Server appliance 5.0 
  1. Start a SSH session to the vCenter Server Appliance. When you are prompted to log in, use the user root and the password:
    1. User: root
    2. Password: vmware (default) 
  1. To query the DB2 database(s), perform one of these options:

Use isql to query the DB2 database.

Note: Make sure you are at the root prompt/user.

  • Retrieve the password for the vCenter Server database user with the command:
    •  grep EMB_DB_PASSWORD /etc/vmware-vpx/embedded_db.cfg
  • Using the retrieved password string, run this command to connect via isql:
    •  VCSA-50:~ # isql -v 'VMware VirtualCenter' vc password
  • When connected, you see a SQL prompt that appears similar to SQL>. You can now query the DB2 database from the prompt.  
  • Run the below command to count the number of rows from VPX_EVENT_ARG table
    • VCSA-50:~#select count(*) from VPX_EVENT_ARG
    • VCSA-50:~# Output similar to 1000054
  • Run the below command to delete the contents from VPX_EVENT_ARG table
    • VCSA-50:~#delete from VPX_EVENT_ARG

Note: Sometimes the Transaction log will be full and the delete commands gets terminated, in these scenario you can delete 100000 rows at a time to avoid transactional logs fill-up

    •  VCSA-50:~#delete from (select * from VPX_EVENT_ARG fetch first 100000 rows only)

Note: Use all CAPS for the SQL quries if it don’t work

  • Proceed with the Import from the newly deployed vCSA 5.1
This post first appeared on the Virtualization Express blog at virtualizationexpress.com, by Karthic Kumar +.
Copyright © 2013 – VirtualizationExpress +. All rights reserved. Not to be reproduced for commercial purposes without written permission.

vCloud Startup Issue

vCD service fails to start for numerous reasons, if you understand what exactly it requires to start the vCD service then you may resolve the issues. I tried explaining what happens when you start the vCloud director services. Below is the workflow that I understood from my experience

Start-up Flow:

Startup

vCloud Director uses JAVA OSGI, which is Java Open Services Gateway Initiative (OSGi) defines an architecture for developing and deploying modular applications and libraries.

To know more about OSGI you may refer wiki article here. What we need to understand is, OSGI is a Container or Folder which has its own sub/internal components. OSGI should be completely up and running for vCloud to properly function.

To begin with check the troubleshooting checklist first. Detailed information can be found below.

Verifying IP/FDQN:

Check if the RHEL cell is configured with proper FQDN and make sure point to a valid DNS address. This is the initial requirement, 2 IPs, one for vCloud HTTP traffic and 2nd one for console proxy, of-course it wouldn’t allow you to install vCloud director application if you do not have 2 nics / ips. There may be chances that NIC might have removed by mistake

 Certificate:

SSL certificate is a crucial requirement for vCloud director since it is a enterprise application and it is used largely by public service providers. The customers / clients / end-user should be confident that their connectivity is secured and encrypted so that a intruder cannot access the packets.Check if the certificates are valid and updated. you may run the below command in the vCloud director server (RHEL / Centos with vCloud Installed).

#opt/vmware/vcloud-director/jre/bin/keytool -storetype JCEKS -storepass <em>passwd</em> -keystore certificates.ks -list

To know more about creating / validating certificates, check below VMware KB.

 

http://kb.vmware.com/kb/1026309

NTP Issues:
Make sure all the  below components are properly synced with NTP server.
  • vCenter Server
  • vShield Manager
  • All vCloud Cells
  • vCloud DB
 DB Connectivity  Issues:
vCD stores all its information in the database, the main reason behind this is clustering and load balancing. you can have more than one vCloud director cell for load balancing and faster access. So if you have more than one vCloud cell then all the cells must have proper connectivity to the Database. At present vCD support Oracle and MSSQL, check the vmware interop matrix for supported versions.Verify the vCD cell can communicate with DB server, may be try a basic test.To know the configured DB for vCloud, you may  check in the global.properties file, there are lot of information in global.properties but we need only the database properties, you may run below command
#grep database.jdbcUrl /opt/vmware/vcloud-director/etc/global.properties

After you get the DB FQDN /IP, perform telnet with respective port to check the connectivity 

#telnet <DB_SERVER> 1433 (MSSQL)
#telnet <DB_SERVER> 1521 (Oracle)

Sometimes vCD can connect to the DB server but it is not enough, the user must have DB_Owner privileges to the vCloud Database. if there is permission issue for the DB user, the OSGI container may start but you cannot access the vCloud UI, it shows a blank grey screen. To find such errors, you can verify cell.log for detailed information.

Reviewing vCloud cell.log
Sometimes the vCD services are fully started but you may see a blank/black screen in the vCD GUI, the first place to look vCD startup issue is
$VCLOUD_HOME/logs/cell.log.

Below is an example of a fully started cell.log file so you can use it to compare.  This log re-writes every time the application restart, you may use this as reference.

 

Application startup begins: MM/DD/YY HH:MM AM
Successfully bound network port: 80 on host address: 192.168.110.xxx
Successfully bound network port: 443 on host address: 192.168.110.xxx
Application Initialization: 9% complete. Subsystem 'com.vmware.vcloud.common.core' started
Successfully connected to database: jdbc:oracle:thin:@Oracle01.test.local:1521/orcl
Successfully bound network port: 443 on host address: 192.168.110.yyy
Successfully bound network port: 61616 on host address: 192.168.110.xxx
Successfully bound network port: 61613 on host address: 192.168.110.xxx
Application Initialization: 18% complete. Subsystem 'com.vmware.vcloud.common-util' started
Application Initialization: 27% complete. Subsystem 'com.vmware.vcloud.consoleproxy' started
Application Initialization: 36% complete. Subsystem 'com.vmware.vcloud.vlsi-core' started
Application Initialization: 45% complete. Subsystem 'com.vmware.vcloud.vim-proxy' started
Successfully verified transfer spooling area: /opt/vmware/cloud-director/data/transfer
Application Initialization: 54% complete. Subsystem 'com.vmware.vcloud.backend-core' started
Application Initialization: 63% complete. Subsystem 'com.vmware.vcloud.imagetransfer-server' started
Application Initialization: 72% complete. Subsystem 'com.vmware.vcloud.rest-api-handlers' started
Application Initialization: 81% complete. Subsystem 'com.vmware.vcloud.ui.configuration' started
Application Initialization: 90% complete. Subsystem 'com.vmware.vcloud.jax-rs-servlet' started
Application Initialization: 100% complete. Subsystem 'com.vmware.vcloud.ui-vcloud-webapp' started
Application Initialization: Complete. Server is ready in 0:46 (minutes:seconds)
Successfully initialized ConfigurationService session factory
Successfully started scheduler
Successfully started remote JMX connector on port 8999

vCloud Director Troubleshooting.

I started working with vCloud director from its initial release (vCloud 1.0) and I was thinking to write an article about the vCloud director troubleshooting for very long time but finally vCloud director(vCD) is going to die oops..I.e it is getting better and better , have a look at below articles
 
 
still it’s not late to write, as per the VMware KB2057589 support for vCD 5.5 is available till Q3 2017 and vCD 5.1 till Sep 2014.
 

What is in this post ?

Troubleshooting , resolving vCloud director issues are complex and tricky, there are no straight forward method to resolve those issues (For Ex:Go ahead just “Restart the service”, “Reboot the server” won’t work with vCloud director) . It needs more  patience and calm approach. To understand the behavior, it is important to understand all the components / services in the vCloud. I tried explaining my way of  approach, below are the topics I covered which I feel it is essential to read.

 
 

 

vCD & its components

You must understand what is vCloud director, what it can do, how it can be used to implement IaaS (Infrastructure as a Service) , there are numerous blog-post and references explaining what is vCloud director (vCD), some of them which I think it is important to read  are listed below.

 

vCloud Training :

 
I hope the above links will make clear about vCD, I would like to cover troubleshooting part of vCD, simply I listed the software components that makes vCD which is required for troubleshooting.
 

Components:

These are the collective software components which makes vCloud 
vCloud director
vCloud Networking and Security (Formerly vShield Manager)
vCenter Server
ESX(i)
vCenter Chargeback manager (optional)

 

vCD Cell Architecture:

 
To begin with vCD troubleshooting, one must understand its Architecture and components. I have summarized the component architecture and how it is connected.
Readers should keep this architecture diagram in mind. To approach an issue in vCD, for example if you are not able to view the vCloud director UI, it should strike that JAVA OSGI should be completely UP and running to access the UI, likewise the below diagram explains the internal connectivity and its relationship.
CellArchitechture
 
 
I have listed only the required components for troubleshooting, to know more about the vCD architechture review the below VMware KB.
 
Internal Components of vCD:
 

vCloud Cell is a Supported Linux Server with vCloud director software installed.

 

Transfer Service is involved when a user in vCloud director uploads a OVF / Media to their allocated resources

 

Console Proxy is proxy service for VMRC- Virtual Machine Remote Console, this component re-directs the VMRC from the ESXi host to vCloud director UI.

 

VC Proxy / Cell Listener service  is agent which sends / receives updates from the vCenter server, like power on/off VM, Create VM…Etc…

 

Active MQ – Message Bus is communication bus between multiple vCloud cells, it is for synchronization.


vCD log components:

 
If you encounter any issues in vCD you must review the logs  to find the cause and resolve the issue.
 
Login to your vCD cell via SSH (Often Putty) and navigate to vCD installation directory i.e /opt/vmware/vcloud-director/
 
Note: $VCLOUD_HOME is the environment variable for the vCloud director installation folder, use this to go to the location.
 
Example cd $VCLOUD_HOME/logs
 
These are the logs that will be useful in reviewing the vCD issues.

cell.log Start up log.
vcloud-container-debug.log  Debugging log messages from the cell
vcloud-container-info.log Warnings or errors in the cell
vmware-vcd-watchdog.log Log of the Watchdog service – which restarts the vCD service if it hangs or stops
 
Required services for vCloud director
  • vCenter services
  • VMware SSO services (Including all dependent services) (Applies on vCenter 5.1 and above)
  • VMware vSphere Profile-Driven storage Service  (Applies on vCenter 5.1 and above)
  • vShield Manager
  • vCloud services (vmware-vcd)

vCD Troubleshooting Checklist:

These are the common items must take into consideration before we begin troubleshooting

vCD Components:
  • vCenter and its DB
  • vCloud and its DB
  • vShield Manager
Check List:
  • Time Sync – Check if the time is synchronized of all vCD components
  • Name Resolution – Check if the vCD components are resolved by its FQDN
  • Valid SSL certificate on all vCD components
  • Connectivity between all vCD components
    • Perform Ping, telnet, netcat test to validate
  • Synchronization – Check if the synchronization of all vCD components
    • In vCloud director –> System tab –> vSphere resources –> vCenter, see if there is tick mark in the Status bar
    • In vCloud director –> System tab –> Cloud Resources –> Cells, check if the vCenter proxy is running
    • In vShield manager –> Configuration — >vCenter, Check the last inventory sync, it should contain the latest time stamp

 

vCD Issues:

 
Below listed issues are most common issues I have seen in vCloud Director.
  • Service Start-up Issue
  • VMRC Issues
  • Ovf Upload issue
  • Networking issues ..ahem
  • Resource allocation issues
  • Multi Cell and Load balancer
  • Guest Customization
  • Connectivity issues
    • vCenter
    • vShield
PS: I have published only the vCD startup issue, I am working on others and will publish shortly. 
 
This post first appeared on the Virtualization Express blog at virtualizationexpress.com, by Karthic Kumar +.
Copyright © 2013 – VirtualizationExpress +. All rights reserved. Not to be reproduced for commercial purposes without written permission.

How to backup and restore of vCloud Usage Meter DB

Backup / Restore of vCloud Usage Meter DB

In this post, I explained how to backup/restore the vCloud Usage meter DB that I found through surfing, collectively it took me more than 60+ minutes to find the steps.

Any one can use this post regardless of the Usage Meter version.

 

To Backup: 


– Login to Usage Meter appliance as root  and run following commands

#su - postgres 



#cd /opt/vmware/vpostgres/xx/bin/


(xx = Version of the postgres, it varies version to version)

#./pg_dump usage_meter_db >> /tmp/dbbackup


To Restore: 

Take a snapshot of the VM before restoring

Login to Usage Meter appliance as root  and run following commands

su - postgres 



cd /opt/vmware/vpostgres/xx/bin/


(xx = Version of the postgres, it varies version to version)

./dropdb usage_meter_db


(Delete the existing DB

./createdb usage_meter_db


(Create a empty DB)

psql usage_meter_db < /tmp/dbbackup


(Restore DB)

or

sudo -u postgres </span>/opt/vmware/vpostgres/xx/bin/psql usage_meter_db &lt; /tmp/dbbackup<span> usage_meter_db


(xx = Version of the postgres, it varies version to version)

 

This post first appeared on the Virtualization Express blog at virtualizationexpress.com, by Karthic Kumar +.
Copyright © 2013 – VirtualizationExpress +. All rights reserved. Not to be reproduced for commercial purposes without written permission.