Thursday, April 11, 2013

VMware VirtualCenter Server - high memory utilization in Java

This applies to vSphere vCenter Server version 5.1.x but may pertain to other versions.

I didn't find much info out there when Goggling so I'm adding what I found here.  I searched for:
vCenter server high memory utilization
VMware vCenter java high memory
VirtualCenter Server out of memory
VMWare vCenter Inventory Service memory

We upgraded our vCenter server to  5.1.0-947939, after which the server (with 4GB of RAM) was starving for memory.  There were 5 instances of JAVA.EXE, each eating up a lot of RAM.  The worst single instance of JAVA was using 1.5GB of RAM, the next one was nearly at 1GB.

In my searches I eventually found someone that was able to tweak the Java Heap Size default.  There are many vCenter Server services that use Tomcat and Java, when you add them all up... ouch, it consumes all the RAM and brings the server to a grind.

What I found:
The main culprit was the "VMWare vCenter Inventory Service."  This had the 1.5GB Java process.  I found you could actually shut it down and the vSphere Client would still work.  I won't pretend to know exactly what this service is for; but it seems to store the client INVENTORY data in a memory cache.  I have only about 30 VMs and 3 hosts, hardly needing 1.5GB of cached info I would guess.

In addition to the Inventory Service, I found you could shut down the "VMware VirtualCenter Managment Webservices" also; however, this will cause an IE type error message in the vSphere Client if you try to look at OVERVIEW on the PERFORMANCE tab.

If you don't want to use the web-client, you can shut down "VMware vSphere Web Client" for a big chunk of memory also.  Then if you do that, you also don't need "VMwareVCMSDS" which provides LDAP services (but it doesn't use much memory.)  If you want to use the Web Client, you do need the LDAP service running though.

The other services are as such:
  • VMware Log Browser: not needed if you're not using the web client.  Not a huge hog though.
  • VMware USB Arbitration Service: not a huge hog.  
  • VMware vCenter Orchestrator Configuration:  I don't use this, I have it disabled so I don't know what memory impact it has. 
  • VMware VirtualCenter Server:  The main service - you need this! 
  • vCenter Single Sign On - does eat up some memory, but is nice to have.
  • VMware vSphere Profile-Driven Storage Service - I'm not really sure what this does to be honest.  I turned it off to no noticable issues.  
THE FIX
If you want the vCenter Inventory Service, and the Web Client running, there is a way to trim down the memory that JAVA uses.  You can tweak config files (wrapper.conf) on the vCenter Server.   Again, the biggest hog is the Inventory Service.  Here is how to reduce the memory requirements. 
  1. Go to: C:\Program Files\VMware\Infrastructure\Inventory Service\conf
  2. Save a backup copy of WRAPPER.CONF
  3. Edit WRAPPER.CONF with a text editor
  4. Look for the section: # Maximum Java Heap Size (in MB)
  5. Modify "wrapper.java.maxmemory"  Mine was set to 3072, I changed mine to 384.  
  6. Save the file.
  7. Open Task Manager if you wish, and look for the JAVA process using the most RAM.
  8. Shutdown / Restart the service: VMware vCenter Inventory Service.
  9. Observe the memory difference.  
It didn't comply 100% to the setting for me, but it did keep it to around 500MB in my case.  

You can also tweak the other services the same way.
  • Webclient - Edit: C:\Program Files\VMware\Infrastructure\vSphereWebClient\server\bin\service\conf\wrapper.conf
  • Management Webservices - Edit: C:\Program Files\VMware\Infrastructure\tomcat\conf\wrapper.conf
  • Log Browser - Edit: C:\Program Files\VMware\Infrastructure\vSphereWebClient\logbrowser\conf\wrapper.conf
If the MAXMEMORY value isn't there you can add it.

I kept all the services running and just tweaked the wrapper.conf files.  My server went from 3.95GB of ram used (and running like crap) to 2.8GB used and running pretty good. 

If you don't have many hosts and VMs you could have it running lean and mean (under 1GB of RAM) by just running the services below.  This may be at the expense of vSphere Client performance and not seeing overview performance charts. 
  • vCenter Single Sign On
  • VMware USB Arbitration Service
  • VMware VirtualCenter Server
  • I also SQL on this box.  

18 comments:

  1. Just wanted to say thank you. This helped me trim the fat a tad on our bogged down VCenter box.

    ReplyDelete
  2. Fantastic! My vCenter was getting so bogged down I was seeing host disconnects due to timeouts, and a 5.8GB RAM (7GB commit) usage is now down to 3.8GB RAM (5GB commit).

    ReplyDelete
    Replies
    1. Good to hear. I'm glad my research helped someone else as well!

      Delete
  3. vCenter Single Sign On - does eat up some memory, but is nice to have.

    Its nice to have.. its a requirement of vCenter after 5.1.

    ReplyDelete
  4. Thank you ! It works perfectly !

    ReplyDelete
  5. How is your vSphere Client performing? We found it so slow that we actually increased the Java heap size. Our vCenter VM is now 32GB RAM, with the biggest java.exe process regularly consuming 12GB+. Lowering the heap size is great for reducing RAM usage on the server itself, but will often come at the expense of client application performance.

    ReplyDelete
  6. It's working fine as configured. We only have 3 hosts and about 30 VMs.

    ReplyDelete
  7. Just want to echo these guys and say thanks, you saved me a lot of time.

    ReplyDelete
  8. Hi,
    I have changed the wrapper.conf as suggested and now CPU spike is went down,but the VMware vCenter Inventory Service is failed to start.
    Please suggest what to do now.

    ReplyDelete
  9. Thanks, this was very helpful when I was troubleshooting high memory usage in VCenter.

    ReplyDelete
  10. Here is the official version direct from VMware support - http://kb.vmware.com/kb/2021302

    ReplyDelete
    Replies
    1. ha... I'm glad they finally addressed it...

      Delete
  11. One to watch out for is enabling VM Flash Cache in 5.5 U1. This can exhibit the same symptoms, high memory and CPU use by the JAVA SE processes but changing the memory limits does nothing to help. Link enclosed.

    http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2072392

    ReplyDelete
  12. In my case vcenter single sign on was taking 2,8G. restart. cleaned.
    thanks

    ReplyDelete
  13. Some times i just think what will be some customers life without help like this one found on internet.
    The softwares producer have take more carefull about their products and "new features".
    I read a lot of tech material from vmware before upgrade and after, i have lost about 8 hours trying
    to figure out my vcenter java memory problem, than i found your site.
    I use some of your sugestions and like magic my memory that was 97% (3,97 gb) now is now 54%.

    This is not a nice post, this is an amazing post.

    Many thanks Brian Kayser for your technotes.

    Thanks for share.

    Márcio from brazil

    ReplyDelete

Please let me know if this helped you out, or if you would like to submit other suggestions or correct something I may have mis-stated.

About Me

My photo
Science Fiction Author / Vice President of Technology for The Christman Company