vCenter Appliance upgrade failed

I was trying to upgrade vCenter Server Appliance 5.0 to vCSA 5.1, there is no inplace upgrade since

 vCSA 5.0 uses IBM DB2 as a DB and

vCSA 5.1 use VMware vPostgres DB ,

so the only available option is migrate the DB

Steps for Upgrade

Keep the vCSA 5.0 Powered on –> Deploy new vCSA 5.1 –> Enable the Trust between both Appliance –> Start Importing.

Detailed steps / videos can be found on below links.

Error that I faced while upgrading was with the iNodes on the new appliance, there were no free iNodes left so that Upgrade failed with error “General system error occurred”

While reviewing the upgrade log /var/log/vmware/vami/upgrade.log there are more than 1 million (10 Lakhs) records on VPX_EVENT_ARG, all the vC event descriptions are stored in this table and it was the cause to fill-up the iNodes .

~ # df -ih 
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sda3   632K  632K   0     100%   /
devtmpfs 0 0 0 - /dev

iNodes are nothing but Index-node to represent the filesystem. for each file it takes each iNode for its ease of accessibility. since it has 1 million records it failed to Index all the files. I was thinking of increasing the iNodes but to increase iNodes it is a requirement to re-create the filesystem, so I approached alternative plan to delete the contents from VPX_EVENT_ARG table.

It was difficult for me to find the commands to truncate / delete the DB. I used below commands to delete the contents from VPX_EVENT_ARG table.

 

  1. Backup and Snapshot the vCenter Server appliance 5.0 
  1. Start a SSH session to the vCenter Server Appliance. When you are prompted to log in, use the user root and the password:
    1. User: root
    2. Password: vmware (default) 
  1. To query the DB2 database(s), perform one of these options:

Use isql to query the DB2 database.

Note: Make sure you are at the root prompt/user.

  • Retrieve the password for the vCenter Server database user with the command:
    •  grep EMB_DB_PASSWORD /etc/vmware-vpx/embedded_db.cfg
  • Using the retrieved password string, run this command to connect via isql:
    •  VCSA-50:~ # isql -v 'VMware VirtualCenter' vc password
  • When connected, you see a SQL prompt that appears similar to SQL>. You can now query the DB2 database from the prompt.  
  • Run the below command to count the number of rows from VPX_EVENT_ARG table
    • VCSA-50:~#select count(*) from VPX_EVENT_ARG
    • VCSA-50:~# Output similar to 1000054
  • Run the below command to delete the contents from VPX_EVENT_ARG table
    • VCSA-50:~#delete from VPX_EVENT_ARG

Note: Sometimes the Transaction log will be full and the delete commands gets terminated, in these scenario you can delete 100000 rows at a time to avoid transactional logs fill-up

    •  VCSA-50:~#delete from (select * from VPX_EVENT_ARG fetch first 100000 rows only)

Note: Use all CAPS for the SQL quries if it don’t work

  • Proceed with the Import from the newly deployed vCSA 5.1
This post first appeared on the Virtualization Express blog at virtualizationexpress.com, by Karthic Kumar +.
Copyright © 2013 – VirtualizationExpress +. All rights reserved. Not to be reproduced for commercial purposes without written permission.

VMware vSphere ESXi Commands for everyday use

In this post, I tried listing down all commands that I use everyday for ESXi Troubleshooting. I have categorized the commands for easy accessibility.

I find these commands are very useful when the ESXi host disconnected from vCenter and when not able to connect to vSphere Client.

[toggle type=”first” title=”Virtual Machine Operations”]

To get All registered VMs on the Host:
vim-cmd vmsvc/getallvms

Example:

~ # vim-cmd vmsvc/getallvms
Vmid Name     File                                  Guest OS Version       Annotation
1    DeskTop  [datastore1 (3)] DeskTop/DeskTop.vmx  windows8_64Guest       vmx-08
2    Domain   [datastore1 (3)] Domain/Domain.vmx    windows7Server64Guest  vmx-08
3    VCD55    [datastore1 (3)] VCD55/VCD55.vmx      centos64Guest          vmx-08
Power ON a VM:
vim-cmd vmsvc/power.on <VMID> 

Example:

~ # vim-cmd vmsvc/power.on 3
Powering on VM:
Power OFF a VM:
vim-cmd vmsvc/power.off <VMID> 

Example:

~ # vim-cmd vmsvc/power.off 3
Powering off VM:
 Gracefully shutdown a VM’s Guest OS:
vim-cmd vmsvc/power.shutdown <VMID> 

Example:

~ # vim-cmd vmsvc/power.shutdown 3
~ #
Register a VM
vim-cmd solo/registervm <vmxpath> 

Example:

~ # vim-cmd solo/registervm "/vmfs/volumes/datastore1 (3)/VCD55/VCD55.vmx"
4

UnRegister a VM:
vim-cmd vmsvc/unregister <VMID> 

Example:

~ # vim-cmd vmsvc/unregister 4
~ #

 

[/toggle]

[toggle title=”Host Operations”]

Enter Host into Maintenance mode:
 vim-cmd hostsvc/maintenance_mode_enter  

 

Exit Host into Maintenance mode:

 vim-cmd hostsvc/maintenance_mode_exit  

 

[/toggle]

[toggle title=”Storage”]

 

Storage Commands

 

LUNs

To get All disks attached to ESXi host.
 esxcfg-scsidevs -l 

or

 esxcli core storage device list 

Sample Output.

naa.600508b1001c0f0e831036c264b4203e
 Display Name: HP Serial Attached SCSI Disk (naa.600508b1001c0f0e831036c264b4203e)
 Has Settable Display Name: true
 Size: 429215
 Device Type: Direct-Access
 Multipath Plugin: NMP
 Devfs Path: /vmfs/devices/disks/naa.600508b1001c0f0e831036c264b4203e
 Vendor: HP
 Model: LOGICAL VOLUME
 Revision: 3.54
 SCSI Level: 5
 Is Pseudo: false
 Status: degraded
 Is RDM Capable: true
 Is Local: false
 Is Removable: false
 Is SSD: false
 Is Offline: false
 Is Perennially Reserved: false
 Queue Full Sample Size: 0
 Queue Full Threshold: 0
 Thin Provisioning Status: unknown
 Attached Filters:
 VAAI Status: unknown
 Other UIDs: vml.0200010000600508b1001c0f0e831036c264b4203e4c4f47494341
 Is Local SAS Device: false
 Is Boot USB Device: false
 No of outstanding IOs with competing worlds: 32

 

 To get All disks attached to ESXi host(Compact List)
~ # esxcfg-scsidevs -c
Device UID Device Type Console Device Size Multipath PluginDisplay Name
naa.600508b1001c0f0e831036c264b4203e Direct-Access /vmfs/devices/disks/naa.600508b1001c0f0e831036c264b4203e 429215MB NMP HP Serial Attached SCSI Disk (naa.600508b1001c0f0e831036c264b4203e)
LUN to Datastore Mapping
~ # esxcfg-scsidevs -m
naa.600508b1001c0f0e831036c264b4203e:3 /vmfs/devices/disks/naa.600508b1001c0f0e831036c264b4203e:3 5131890d-0c77f39a-0382-e4115b9b7270 0 datastore1 (3)

LUN Paths:

 

~ # esxcfg-mpath -L
 vmhba0:C0:T0:L1 state:active naa.600508b1001c0f0e831036c264b4203e vmhba0 0 0 1 NMP active san sas.5001438027641890 sas.1438027641890

 ~ # esxcfg-mpath -m
 vmhba0:C0:T0:L1 vmhba0 sas.5001438027641890 sas.1438027641890 naa.600508b1001c0f0e831036c264b4203e

 ~ # esxcfg-mpath -b
 naa.600508b1001c0f0e831036c264b4203e : HP Serial Attached SCSI Disk (naa.600508b1001c0f0e831036c264b4203e)
 vmhba0:C0:T0:L1 LUN:1 state:active sas Adapter: 5001438027641890 Target: 1438027641890

 

HBA Commands

To List all HBAs available on the ESXi Host.
~ # esxcfg-scsidevs -a
vmhba0 hpsa link-n/a sas.5001438027641890 (0:3:0.0) Hewlett-Packard Company Smart Array P220i
vmhba1 qlnativefc link-up fc.50060b0000c26205:50060b0000c26204 (0:5:0.0) QLogic Corp ISP2532-based 8Gb Fibre Channel to PCI Express HBA
vmhba2 qlnativefc link-n/a fc.50060b0000c26207:50060b0000c26206 (0:5:0.1) QLogic Corp ISP2532-based 8Gb Fibre Channel to PCI Express HBA
vmhba32 iscsi_vmk online iscsi.vmhba32 iSCSI Software Adapter
HBA and LUN Mapping.
~ # esxcfg-scsidevs -A
vmhba0 naa.600508b1001c0f0e831036c264b4203e
To Rescan a HBA
esxcfg-rescan <HBANAME>

example

~ # esxcfg-rescan vmhba32
~ #

iSCSI Commands

Discovery Stats for a iSCSI adapter

~ # vmkiscsi-tool -D <HBANAME>

Example

~ # vmkiscsi-tool -D vmhba32
=========Discovery Properties for Adapter vmhba32=========
 iSnsDiscoverySettable : 0
 iSnsDiscoveryEnabled : 0
 iSnsDiscoveryMethod : 0
 iSnsHost.ipAddress : ::
 staticDiscoverySettable : 0
 staticDiscoveryEnabled : 1
 sendTargetsDiscoverySettable : 0
 sendTargetsDiscoveryEnabled : 1
 slpDiscoverySettable : 0
 No Discovery Addresses Found
 No Static Discovery Targets Found
~ # vmkiscsi-tool -E <HBANME>

Example

 ~ # vmkiscsi-tool -E vmhba32
 No active targets.
ISCSI – IQN Name
~ # vmkiscsi-tool -I HBANAME

Example

~ # vmkiscsi-tool -I vmhba32
iSCSI Node Name: iqn.1998-01.com.vmware:is-tse-h44-56726a3c

MTU Value
~ # vmkiscsi-tool -M vmhba32

Example.

 ~ # vmkiscsi-tool -M vmhba32
 MTU SETTABLE : Not Valid
 MTU Size : Not Valid

 

NFS Commands

To list all the mounted NFS datastores.
~ # esxcli storage nfs list

or

~ # esxcfg-nas -l

 

Add a NFS Datastore
~ # esxcli storage nfs add -H NFS_IP|NFS_HOSTNAME -s Share_mount_point_on_the_NFS -vDatastoreName

or

~ # esxcfg-nas -a -o NFS_IP -s Share_mount_point_on_the_NFSDatastoreName
Delete a NFS Datastore
 esxcli storage nfs remove -v NFS_Datastore_Name

or

esxcfg-nas -d NFS_Datastore_Name

 

 

[/toggle]

[toggle title=”Networking”]Still Working on [/toggle]

[toggle title=”vSphere Replication”]

vim-cmd hbrsvc/

[/toggle]

 

[toggle title=”Advanced & Misc “]Working on[/toggle]

Usage Meter 3.2 Import data fails

Look at the old post for How to upgrade UM to 3.2 . In this post I will cover the error faced while upgrade.

When I was  importing the data from old appliance, I got below error

Connection closed by remote host

To dig this error in detail, I trid to ssh to old appliance

ssh root@old-appliance-ip

 

I got error "Connection closed by remote host", this didn’t give me much information why was the error occurred, so thought of getting more information to resolve the issue. I tried ssh with verbose logging so that I can see what exactly happening in the background, tried with below command and it did give me detailed information about the issue.

ssh -vv root@old-appliance-ip
debug2: ssh_connect: needpriv 0
debug1: Connecting to localhost [xx.xx.xx.xx] port 22.
debug1: Connection established.
debug3: Not a RSA1 key file /c/Users/MoreFreeze/.ssh/id_rsa.
debug2: key_type_from_name: unknown key type '-----BEGIN'
debug3: key_read: missing keytype
debug3: key_read: missing whitespace 
// above it repeats 24 times
debug2: key_type_from_name: unknown key type '-----END'
debug3: key_read: missing keytype
debug1: identity file /c/Users/MoreFreeze/.ssh/id_rsa type 1
debug1: identity file /c/Users/MoreFreeze/.ssh/id_dsa type -1
ssh_exchange_identification: Connection closed by remote host

 

This proves that there is some error and it was confusing me with the certificate, after 15-20 minutes of googling found that there is something wrong in the hosts.deny file in the /etc/ directory, actually it is not wrong but for security. I edited the hosts.deny file and commented out the line “ALL:ALL”, it started working but I knew that it is not the right way to deal with this, further researching about this gave me idea to add SSHD:ALL in the hosts.allow file in /etc/ directory.

 

  • edit the file /etc/hosts.allow using VI editor, to know how to use VI editor check this VMware KB 1020302
  • vi /etc/hosts.allow
  • Add the line SSHD:ALL
  • Restart the SSH Daemon process
  • service sshd restart
  • To verify tried ssh again, got the password prompt, huh it started working
  • Then tried importum command and the UM upgraded to 3.2 successfully.

 

Please feel free to comment below if you [readers] have faced some other issue.

 

This post first appeared on the Virtualization Express blog at virtualizationexpress.com, by Karthic Kumar .
Copyright © 2013 – VirtualizationExpress . All rights reserved. Not to be reproduced for commercial purposes without written permission.

Upgrading vCloud Usage Meter 3.2 / 3.3 / 3.5

In this post, I will walk you through how to upgrade to UM3.2 / 3.3.

    1. Upgrading Usage meter 3.2 / 3.3 from Usage Meter 2.3.2 to 3.1.0
    2. Upgrading Usage meter 3.3 from Usage Meter 3.2

 

Upgrading Usage meter 3.2 / 3.3 from Usage Meter 2.3.2 to 3.1.0
UM3.2 supports import data from following versions only, 
 vCloud Usage Meter 2.3.2
 vCloud Usage Meter 3.0.0
 vCloud Usage Meter 3.0.2
 vCloud Usage Meter 3.1.0

.[info title=”Note”]if you are having older version then you must upgrade to supported version and then upgrade to 3.2[/info]

Steps for Import:

Flow:

 
 
UM3.2Upgrade

Detailed Steps:

    1.  Download the new Usage Meter Appliance from here  (3.2) & 3.3:
    2.  Deploy and power on new appliance
    3.  Assign IP address
    4.  Go to Old Appliance and enable SSH
      service &lt;span class="mceItemHidden" data-mce-bogus="1"&gt;&lt;span class="hiddenSpellError" pre="service " data-mce-bogus="1"&gt;sshd&lt;/span&gt;&lt;/span&gt; start
    5.  Go to new appliance and run command
       &lt;span class="mceItemHidden" data-mce-bogus="1"&gt;&lt;span class="hiddenSpellError" pre="" data-mce-bogus="1"&gt;importum&lt;/span&gt;&lt;/span&gt; &amp;amp;lt;IPADDRESS&amp;amp;gt; &amp;amp;lt;OldVersion&amp;amp;gt;
       vCloud Usage Meter 2.3.2 -->  232
       vCloud Usage Meter 3.0.0 --> 300
       vCloud Usage Meter 3.0.2 --> 302
       vCloud Usage Meter 3.1.0 --> 310
       &lt;span class="mceItemHidden" data-mce-bogus="1"&gt;&lt;span class="hiddenSpellError" pre="" data-mce-bogus="1"&gt;importum&lt;/span&gt;&lt;/span&gt; xx.xx.xx.xx 232 
    6.  After successful completion, go to the UM 3.2 console and verify if you have all the data
    7. Power off and delete the old Appliance

 

Upgrading vCloud Usage meter from 3.2 to 3.3:
  1. Download vCloud Usage Meter 3.3 Updater from here
  2. Transfer the patch file, usage-meter-330-update.zip to /root in the appliance
  3. Unzip usage-meter-330-update.zip
  4. Enter cd usage-meter-330-update
  5. Enter ./update-um
  6. Appliance is upgraded to 3.3 GA.
    What to do next

    Monitor results to ensure continuous data collection.

Upgrading vCloud Usage meter from 3.2 / 3.3 to 3.5:
  1.  Download the new Usage Meter Appliance 3.5  from here
  2.  Deploy and power on new appliance
  3.  Assign IP address
  4.  Go to Old Appliance and enable SSH
    service &amp;lt;span class="mceItemHidden" data-mce-bogus="1"&amp;gt;&amp;lt;span class="hiddenSpellError" pre="service " data-mce-bogus="1"&amp;gt;sshd&amp;lt;/span&amp;gt;&amp;lt;/span&amp;gt; start
  5.  Go to new appliance and run command
     &amp;lt;span class="mceItemHidden" data-mce-bogus="1"&amp;gt;&amp;lt;span class="hiddenSpellError" pre="" data-mce-bogus="1"&amp;gt;importum&amp;lt;/span&amp;gt;&amp;lt;/span&amp;gt; &amp;amp;amp;lt;IPADDRESS&amp;amp;amp;gt; &amp;amp;amp;lt;OldVersion&amp;amp;amp;gt;
     vCloud Usage Meter 3.3.3 --> 333
     vCloud Usage Meter 3.4.0 --> 340
  6.  After successful completion, go to the UM 3.2 console and verify if you have all the data
  7. Power off and delete the old Appliance / Keep backup

 

[Revision]
27-Dec-2013 – Initial post
5-June-2014 – Updated instructions for UM 3.3

21 Dec-2016 – Updated instructions for UM 3.5

 

 

This post first appeared on the Virtualization Express blog at virtualizationexpress.com, by Karthic Kumar .
Copyright © 2014 – VirtualizationExpress . All rights reserved. Not to be reproduced for commercial purposes without written permission.

vCloud Startup Issue

vCD service fails to start for numerous reasons, if you understand what exactly it requires to start the vCD service then you may resolve the issues. I tried explaining what happens when you start the vCloud director services. Below is the workflow that I understood from my experience

Start-up Flow:

Startup

vCloud Director uses JAVA OSGI, which is Java Open Services Gateway Initiative (OSGi) defines an architecture for developing and deploying modular applications and libraries.

To know more about OSGI you may refer wiki article here. What we need to understand is, OSGI is a Container or Folder which has its own sub/internal components. OSGI should be completely up and running for vCloud to properly function.

To begin with check the troubleshooting checklist first. Detailed information can be found below.

Verifying IP/FDQN:

Check if the RHEL cell is configured with proper FQDN and make sure point to a valid DNS address. This is the initial requirement, 2 IPs, one for vCloud HTTP traffic and 2nd one for console proxy, of-course it wouldn’t allow you to install vCloud director application if you do not have 2 nics / ips. There may be chances that NIC might have removed by mistake

 Certificate:

SSL certificate is a crucial requirement for vCloud director since it is a enterprise application and it is used largely by public service providers. The customers / clients / end-user should be confident that their connectivity is secured and encrypted so that a intruder cannot access the packets.Check if the certificates are valid and updated. you may run the below command in the vCloud director server (RHEL / Centos with vCloud Installed).

#opt/vmware/vcloud-director/jre/bin/keytool -storetype JCEKS -storepass <em>passwd</em> -keystore certificates.ks -list

To know more about creating / validating certificates, check below VMware KB.

 

http://kb.vmware.com/kb/1026309

NTP Issues:
Make sure all the  below components are properly synced with NTP server.
  • vCenter Server
  • vShield Manager
  • All vCloud Cells
  • vCloud DB
 DB Connectivity  Issues:
vCD stores all its information in the database, the main reason behind this is clustering and load balancing. you can have more than one vCloud director cell for load balancing and faster access. So if you have more than one vCloud cell then all the cells must have proper connectivity to the Database. At present vCD support Oracle and MSSQL, check the vmware interop matrix for supported versions.Verify the vCD cell can communicate with DB server, may be try a basic test.To know the configured DB for vCloud, you may  check in the global.properties file, there are lot of information in global.properties but we need only the database properties, you may run below command
#grep database.jdbcUrl /opt/vmware/vcloud-director/etc/global.properties

After you get the DB FQDN /IP, perform telnet with respective port to check the connectivity 

#telnet <DB_SERVER> 1433 (MSSQL)
#telnet <DB_SERVER> 1521 (Oracle)

Sometimes vCD can connect to the DB server but it is not enough, the user must have DB_Owner privileges to the vCloud Database. if there is permission issue for the DB user, the OSGI container may start but you cannot access the vCloud UI, it shows a blank grey screen. To find such errors, you can verify cell.log for detailed information.

Reviewing vCloud cell.log
Sometimes the vCD services are fully started but you may see a blank/black screen in the vCD GUI, the first place to look vCD startup issue is
$VCLOUD_HOME/logs/cell.log.

Below is an example of a fully started cell.log file so you can use it to compare.  This log re-writes every time the application restart, you may use this as reference.

 

Application startup begins: MM/DD/YY HH:MM AM
Successfully bound network port: 80 on host address: 192.168.110.xxx
Successfully bound network port: 443 on host address: 192.168.110.xxx
Application Initialization: 9% complete. Subsystem 'com.vmware.vcloud.common.core' started
Successfully connected to database: jdbc:oracle:thin:@Oracle01.test.local:1521/orcl
Successfully bound network port: 443 on host address: 192.168.110.yyy
Successfully bound network port: 61616 on host address: 192.168.110.xxx
Successfully bound network port: 61613 on host address: 192.168.110.xxx
Application Initialization: 18% complete. Subsystem 'com.vmware.vcloud.common-util' started
Application Initialization: 27% complete. Subsystem 'com.vmware.vcloud.consoleproxy' started
Application Initialization: 36% complete. Subsystem 'com.vmware.vcloud.vlsi-core' started
Application Initialization: 45% complete. Subsystem 'com.vmware.vcloud.vim-proxy' started
Successfully verified transfer spooling area: /opt/vmware/cloud-director/data/transfer
Application Initialization: 54% complete. Subsystem 'com.vmware.vcloud.backend-core' started
Application Initialization: 63% complete. Subsystem 'com.vmware.vcloud.imagetransfer-server' started
Application Initialization: 72% complete. Subsystem 'com.vmware.vcloud.rest-api-handlers' started
Application Initialization: 81% complete. Subsystem 'com.vmware.vcloud.ui.configuration' started
Application Initialization: 90% complete. Subsystem 'com.vmware.vcloud.jax-rs-servlet' started
Application Initialization: 100% complete. Subsystem 'com.vmware.vcloud.ui-vcloud-webapp' started
Application Initialization: Complete. Server is ready in 0:46 (minutes:seconds)
Successfully initialized ConfigurationService session factory
Successfully started scheduler
Successfully started remote JMX connector on port 8999

vCloud Director Troubleshooting.

I started working with vCloud director from its initial release (vCloud 1.0) and I was thinking to write an article about the vCloud director troubleshooting for very long time but finally vCloud director(vCD) is going to die oops..I.e it is getting better and better , have a look at below articles
 
 
still it’s not late to write, as per the VMware KB2057589 support for vCD 5.5 is available till Q3 2017 and vCD 5.1 till Sep 2014.
 

What is in this post ?

Troubleshooting , resolving vCloud director issues are complex and tricky, there are no straight forward method to resolve those issues (For Ex:Go ahead just “Restart the service”, “Reboot the server” won’t work with vCloud director) . It needs more  patience and calm approach. To understand the behavior, it is important to understand all the components / services in the vCloud. I tried explaining my way of  approach, below are the topics I covered which I feel it is essential to read.

 
 

 

vCD & its components

You must understand what is vCloud director, what it can do, how it can be used to implement IaaS (Infrastructure as a Service) , there are numerous blog-post and references explaining what is vCloud director (vCD), some of them which I think it is important to read  are listed below.

 

vCloud Training :

 
I hope the above links will make clear about vCD, I would like to cover troubleshooting part of vCD, simply I listed the software components that makes vCD which is required for troubleshooting.
 

Components:

These are the collective software components which makes vCloud 
vCloud director
vCloud Networking and Security (Formerly vShield Manager)
vCenter Server
ESX(i)
vCenter Chargeback manager (optional)

 

vCD Cell Architecture:

 
To begin with vCD troubleshooting, one must understand its Architecture and components. I have summarized the component architecture and how it is connected.
Readers should keep this architecture diagram in mind. To approach an issue in vCD, for example if you are not able to view the vCloud director UI, it should strike that JAVA OSGI should be completely UP and running to access the UI, likewise the below diagram explains the internal connectivity and its relationship.
CellArchitechture
 
 
I have listed only the required components for troubleshooting, to know more about the vCD architechture review the below VMware KB.
 
Internal Components of vCD:
 

vCloud Cell is a Supported Linux Server with vCloud director software installed.

 

Transfer Service is involved when a user in vCloud director uploads a OVF / Media to their allocated resources

 

Console Proxy is proxy service for VMRC- Virtual Machine Remote Console, this component re-directs the VMRC from the ESXi host to vCloud director UI.

 

VC Proxy / Cell Listener service  is agent which sends / receives updates from the vCenter server, like power on/off VM, Create VM…Etc…

 

Active MQ – Message Bus is communication bus between multiple vCloud cells, it is for synchronization.


vCD log components:

 
If you encounter any issues in vCD you must review the logs  to find the cause and resolve the issue.
 
Login to your vCD cell via SSH (Often Putty) and navigate to vCD installation directory i.e /opt/vmware/vcloud-director/
 
Note: $VCLOUD_HOME is the environment variable for the vCloud director installation folder, use this to go to the location.
 
Example cd $VCLOUD_HOME/logs
 
These are the logs that will be useful in reviewing the vCD issues.

cell.log Start up log.
vcloud-container-debug.log  Debugging log messages from the cell
vcloud-container-info.log Warnings or errors in the cell
vmware-vcd-watchdog.log Log of the Watchdog service – which restarts the vCD service if it hangs or stops
 
Required services for vCloud director
  • vCenter services
  • VMware SSO services (Including all dependent services) (Applies on vCenter 5.1 and above)
  • VMware vSphere Profile-Driven storage Service  (Applies on vCenter 5.1 and above)
  • vShield Manager
  • vCloud services (vmware-vcd)

vCD Troubleshooting Checklist:

These are the common items must take into consideration before we begin troubleshooting

vCD Components:
  • vCenter and its DB
  • vCloud and its DB
  • vShield Manager
Check List:
  • Time Sync – Check if the time is synchronized of all vCD components
  • Name Resolution – Check if the vCD components are resolved by its FQDN
  • Valid SSL certificate on all vCD components
  • Connectivity between all vCD components
    • Perform Ping, telnet, netcat test to validate
  • Synchronization – Check if the synchronization of all vCD components
    • In vCloud director –> System tab –> vSphere resources –> vCenter, see if there is tick mark in the Status bar
    • In vCloud director –> System tab –> Cloud Resources –> Cells, check if the vCenter proxy is running
    • In vShield manager –> Configuration — >vCenter, Check the last inventory sync, it should contain the latest time stamp

 

vCD Issues:

 
Below listed issues are most common issues I have seen in vCloud Director.
  • Service Start-up Issue
  • VMRC Issues
  • Ovf Upload issue
  • Networking issues ..ahem
  • Resource allocation issues
  • Multi Cell and Load balancer
  • Guest Customization
  • Connectivity issues
    • vCenter
    • vShield
PS: I have published only the vCD startup issue, I am working on others and will publish shortly. 
 
This post first appeared on the Virtualization Express blog at virtualizationexpress.com, by Karthic Kumar +.
Copyright © 2013 – VirtualizationExpress +. All rights reserved. Not to be reproduced for commercial purposes without written permission.