VMware VirtualCenter Server - high memory utilization in Java

This applies to vSphere vCenter Server version 5.1.x but may pertain to other versions.

I didn't find much info out there when Goggling so I'm adding what I found here.  I searched for:
vCenter server high memory utilization
VMware vCenter java high memory
VirtualCenter Server out of memory
VMWare vCenter Inventory Service memory

We upgraded our vCenter server to  5.1.0-947939, after which the server (with 4GB of RAM) was starving for memory.  There were 5 instances of JAVA.EXE, each eating up a lot of RAM.  The worst single instance of JAVA was using 1.5GB of RAM, the next one was nearly at 1GB.

In my searches I eventually found someone that was able to tweak the Java Heap Size default.  There are many vCenter Server services that use Tomcat and Java, when you add them all up... ouch, it consumes all the RAM and brings the server to a grind.

What I found:
The main culprit was the "VMWare vCenter Inventory Service."  This had the 1.5GB Java process.  I found you could actually shut it down and the vSphere Client would still work.  I won't pretend to know exactly what this service is for; but it seems to store the client INVENTORY data in a memory cache.  I have only about 30 VMs and 3 hosts, hardly needing 1.5GB of cached info I would guess.

In addition to the Inventory Service, I found you could shut down the "VMware VirtualCenter Managment Webservices" also; however, this will cause an IE type error message in the vSphere Client if you try to look at OVERVIEW on the PERFORMANCE tab.

If you don't want to use the web-client, you can shut down "VMware vSphere Web Client" for a big chunk of memory also.  Then if you do that, you also don't need "VMwareVCMSDS" which provides LDAP services (but it doesn't use much memory.)  If you want to use the Web Client, you do need the LDAP service running though.

The other services are as such:

• VMware Log Browser: not needed if you're not using the web client.  Not a huge hog though.
• VMware USB Arbitration Service: not a huge hog.  
• VMware vCenter Orchestrator Configuration:  I don't use this, I have it disabled so I don't know what memory impact it has. 
• VMware VirtualCenter Server:  The main service - you need this! 
• vCenter Single Sign On - does eat up some memory, but is nice to have.
• VMware vSphere Profile-Driven Storage Service - I'm not really sure what this does to be honest.  I turned it off to no noticable issues.  

THE FIX

If you want the vCenter Inventory Service, and the Web Client running, there is a way to trim down the memory that JAVA uses.  You can tweak config files (wrapper.conf) on the vCenter Server.   Again, the biggest hog is the Inventory Service.  Here is how to reduce the memory requirements. 

1. Go to: C:\Program Files\VMware\Infrastructure\Inventory Service\conf
2. Save a backup copy of WRAPPER.CONF
3. Edit WRAPPER.CONF with a text editor
4. Look for the section: # Maximum Java Heap Size (in MB)
5. Modify "wrapper.java.maxmemory"  Mine was set to 3072, I changed mine to 384.  
6. Save the file.
7. Open Task Manager if you wish, and look for the JAVA process using the most RAM.
8. Shutdown / Restart the service: VMware vCenter Inventory Service.
9. Observe the memory difference.  

It didn't comply 100% to the setting for me, but it did keep it to around 500MB in my case.  

You can also tweak the other services the same way.

• Webclient - Edit: C:\Program Files\VMware\Infrastructure\vSphereWebClient\server\bin\service\conf\wrapper.conf
• Management Webservices - Edit: C:\Program Files\VMware\Infrastructure\tomcat\conf\wrapper.conf
• Log Browser - Edit: C:\Program Files\VMware\Infrastructure\vSphereWebClient\logbrowser\conf\wrapper.conf

If the MAXMEMORY value isn't there you can add it.

I kept all the services running and just tweaked the wrapper.conf files.  My server went from 3.95GB of ram used (and running like crap) to 2.8GB used and running pretty good. 

If you don't have many hosts and VMs you could have it running lean and mean (under 1GB of RAM) by just running the services below.  This may be at the expense of vSphere Client performance and not seeing overview performance charts. 

• vCenter Single Sign On
• VMware USB Arbitration Service
• VMware VirtualCenter Server
• I also SQL on this box.  

Does the AMD 2356 processor support EVC

Mr. Google couldn't directly answer this question for me so I figured I'd throw it out here just in case someone else wanted the direct answer.  

I had three HP Proliant DL365 G5 servers, with dual Quad-Code AMD Opteron 2356 Processors as my VMware hosts.  They went into production in 2008 and served us well.  I added a new server into the mix, an HP Proliant DL165 G7, with dual AMD Opteron 6220 Processors. 

They did not seem to play well with VMware’s vMotion so I figured I needed to enable the Enhanced vMotion Compatibility (EVC) mode.  Well, the older host didn’t seem compatible with that either.  In vSphere Client, when you look at the SUMMARY of the host, there is a heading “VMware EVC Mode:”  It will say “Disabled”, of course, if it isn’t enabled, but next to it there is a info box.  When you click that it says: “Supported EVC Modes:  EVC Not Supported”.  Drab! 

Not to be discouraged I Googled around but couldn’t find anything that specifically said anything about my older hosts processors supporting EVC or not.  But I did find a VMware tool (and ISO) that creates a boot CD that will tell you for sure what the deal is. 

So, with fingers crossed, I booted-up the old and reliable host with the newly burned bootable CD.  Ah, look at all those YESes!!  

But SNAP – the final nail! “Supported EVC Modes: None”

I guess I’m going server shopping… 

Upgrade Lefthand SAN Disks / HP StorageWorks P4500

Can you upgrade a Lefthand / HP SAN Node's Hard Drives?

According to HP support the answer is YES!!

Here is what I have learned so far.  Yes, you can upgrade your SAN node hard drives simply by replacing the drives with new larger drives and then re-imaging the node with the latest SAN/iQ CD which you can download as an ISO.  

I mainly have P4500 series units.  I have three nodes that have 300GB SCSI drives.  I can replace those with 600GB drives and essentially double my storage space.  I could even go with 1TB or 2TB drives and grow it even larger.  However those are "Midline" drives and you take a 40% performance hit according to HP support.  As of right now the biggest 15K SCSI drives they support are the 600GB drives.  The 1 or 2 TB drives are 7.2K RPM.  To keep HP happy and maintain service contracts you need to use the HP specific drives.  

Another interesting note is that I can even take my SATA SAN and put in SCSI drives to make it a SCSI (SAS) SAN.  Nice.

I'm posting this because I couldn't find anything on-line that even hinted that this was possible, let alone supported.  So from the horses mouth (HP), it is, and if you do it right, it is even supported.  What they do is that if you have a hard drive failure you will have to work with a different department for replacement hard drives.  They have 3 year warranties.  They may not fall under a 4-hour response time (if you have that level), I forgot to ask that.  

That's all I'm going to post right now.  If enough people are interested in the details I'll follow-up with part numbers, the process, etc.  

Let me know by commenting and maybe click an ad or two!  

hard drive upgrade for lefthand SAN   lefthand SAN increase storage space   replace hard drives in HP StorageWorks SAN   how to upgrade Lefthand SAN

how to upgrade HP SAN   Can you put bigger drives in SAN



restripe pending because of cluster edit or migration
data protection degraded