VMware VirtualCenter Server - high memory utilization in Java

This applies to vSphere vCenter Server version 5.1.x but may pertain to other versions.

I didn't find much info out there when Goggling so I'm adding what I found here.  I searched for:
vCenter server high memory utilization
VMware vCenter java high memory
VirtualCenter Server out of memory
VMWare vCenter Inventory Service memory

We upgraded our vCenter server to  5.1.0-947939, after which the server (with 4GB of RAM) was starving for memory.  There were 5 instances of JAVA.EXE, each eating up a lot of RAM.  The worst single instance of JAVA was using 1.5GB of RAM, the next one was nearly at 1GB.

In my searches I eventually found someone that was able to tweak the Java Heap Size default.  There are many vCenter Server services that use Tomcat and Java, when you add them all up... ouch, it consumes all the RAM and brings the server to a grind.

What I found:
The main culprit was the "VMWare vCenter Inventory Service."  This had the 1.5GB Java process.  I found you could actually shut it down and the vSphere Client would still work.  I won't pretend to know exactly what this service is for; but it seems to store the client INVENTORY data in a memory cache.  I have only about 30 VMs and 3 hosts, hardly needing 1.5GB of cached info I would guess.

In addition to the Inventory Service, I found you could shut down the "VMware VirtualCenter Managment Webservices" also; however, this will cause an IE type error message in the vSphere Client if you try to look at OVERVIEW on the PERFORMANCE tab.

If you don't want to use the web-client, you can shut down "VMware vSphere Web Client" for a big chunk of memory also.  Then if you do that, you also don't need "VMwareVCMSDS" which provides LDAP services (but it doesn't use much memory.)  If you want to use the Web Client, you do need the LDAP service running though.

The other services are as such:

• VMware Log Browser: not needed if you're not using the web client.  Not a huge hog though.
• VMware USB Arbitration Service: not a huge hog.  
• VMware vCenter Orchestrator Configuration:  I don't use this, I have it disabled so I don't know what memory impact it has. 
• VMware VirtualCenter Server:  The main service - you need this! 
• vCenter Single Sign On - does eat up some memory, but is nice to have.
• VMware vSphere Profile-Driven Storage Service - I'm not really sure what this does to be honest.  I turned it off to no noticable issues.  

THE FIX

If you want the vCenter Inventory Service, and the Web Client running, there is a way to trim down the memory that JAVA uses.  You can tweak config files (wrapper.conf) on the vCenter Server.   Again, the biggest hog is the Inventory Service.  Here is how to reduce the memory requirements. 

1. Go to: C:\Program Files\VMware\Infrastructure\Inventory Service\conf
2. Save a backup copy of WRAPPER.CONF
3. Edit WRAPPER.CONF with a text editor
4. Look for the section: # Maximum Java Heap Size (in MB)
5. Modify "wrapper.java.maxmemory"  Mine was set to 3072, I changed mine to 384.  
6. Save the file.
7. Open Task Manager if you wish, and look for the JAVA process using the most RAM.
8. Shutdown / Restart the service: VMware vCenter Inventory Service.
9. Observe the memory difference.  

It didn't comply 100% to the setting for me, but it did keep it to around 500MB in my case.  

You can also tweak the other services the same way.

• Webclient - Edit: C:\Program Files\VMware\Infrastructure\vSphereWebClient\server\bin\service\conf\wrapper.conf
• Management Webservices - Edit: C:\Program Files\VMware\Infrastructure\tomcat\conf\wrapper.conf
• Log Browser - Edit: C:\Program Files\VMware\Infrastructure\vSphereWebClient\logbrowser\conf\wrapper.conf

If the MAXMEMORY value isn't there you can add it.

I kept all the services running and just tweaked the wrapper.conf files.  My server went from 3.95GB of ram used (and running like crap) to 2.8GB used and running pretty good. 

If you don't have many hosts and VMs you could have it running lean and mean (under 1GB of RAM) by just running the services below.  This may be at the expense of vSphere Client performance and not seeing overview performance charts. 

• vCenter Single Sign On
• VMware USB Arbitration Service
• VMware VirtualCenter Server
• I also SQL on this box.