cancel
Showing results for 
Search instead for 
Did you mean: 

When Performance Optimization Matters

Level 9

Performance in virtual world is complex. But it's one of the most important topic as well! Every admin I meet puts performance on the first (second) place of his priorities. The other priority is Disaster recovery. Depending of the environments, admins usually prefers to deliver best performance, but cares about fast backups (and recovery) too. No one likes the users complains about performance, right?

Virtualization brought another layer of complexity - Performance bottlenecks.  In full virtual or mixed environments where some shops may also add another complexity with two different hypervizors, things might not be very simple, but rather complex to solve.

Usually there isn't only a single bottleneck as usually bottlenecks are followed with miss-configurations on the VM side. There are some principal miss-configurations that often repeat. Miss-configuration on the VMs side, and also some space waste the infrastructure side as well, which can lead to loss of performance. Here are few areas where to seek for improvements:

  • VMs with multiple vCPU when only single vCPU needed - start with less to add more later.
  • VMs with types of network adapters not adequate for the OS - check the requirements and follow the documentation (RTFM)
  • VMs with too much memory allocation - does that VM really needs 16Gb of RAM?
  • Storage bottlenecks - we all know that storage is the slowest part of the datacenter (except flash). It's changing with server side flash and different acceleration solutions, but also with hyper-converged solutions like VMware VSAN. But the problems might be on the storage network or HBA.

The virtual environment struggles also with the fact that sometimes in smaller shops there aren't much rules on who does what. The case when several admins, plus a developer team takes the virtual infrastructure for their playground. Everyone creates VMs which then sits there doing nothing, VMs snapshots laying around consuming valuable SAN disk space and suddenly there is not enough space on the datastores, backups stops working or VMs are having performance problems. There is a name for that - VM sprawl!

Here also, some order first please. Define rules who does what. Which VMs shall exists and in which situation is important to keep snapshot (or not). Think of archiving VMs rather than creating snapshots. There are some good backup tools for that.

Modern virtualization management tools can help in a situation when suddenly a performance is falling. Some of the tools integrate a possibility to detect configuration changes in the virtual environment. It means that you can see what changed the day that the performance started to be bad. It not always help as there are other factors that falls outside of the scope, and the tools might be monitoring the changes only of the virtual environment itself and not the outside physical world. But it can narrow the problem.

28 Comments
cbussard
Level 10

Over/under allocation of VM's used to be one of my worst enemies. It forced me to learn esxtop really quickly since I work with primarily VMware. I think you hit the nail on the head. CPU contention, over/under allocation of resources, storage bottlenecks, network adapters, vSwitch config, and networking issues.

Also, not to many people out there understand that when you use virtual switches in VMware, all of that processing has to be done somewhere. It is done on your host CPU, which most do not realize, or do not understand.

Great post!

Jfrazier
Level 18

Modern virtualization tools while very useful also have their drawbacks. 

Proper use of tools and the VM engineer checking what they see from one tool against another (different views or perspective) can show a totally different story.

For example, running vCops and using it's recommendations to "rightsize" a server without looking to see that the under-utilization of resources is actually caused by another virtual server hogging the resources thus effectively starving the server that is about to have its environment reduced because it is not using everything.

vladan
Level 9

Sure, true thing. You can have a VM (a "Noisy Neighbor") which pumps all, or most of the resources (IOPS) and all the other VMs on the same datastore are just stuggling and showing high latency. No good to rely only on single tool and better double check via ESXTOP.

Webbster
Level 10

Not being a Storage or VM geek. These topics are extremely useful in expanding my knowledge of technologies.

aaron.j.denning
Level 12

Agree with Webbster dont know alot about VM or Storage but wanting to get into that realm just to have that up my sleeve as a "we need help with" and kinda step in like superman and be like i can do that.

ideal-jmm
Level 7

Good tips. Even at a small business level, improper VM allocation planning can significantly impact prime-time processes and user experiences. I've found that setting a minimum resource quota can allow host processing when servers are at their highest load along with a point to stop and recognize that you may need to address future configuration issues sooner than originally expected. Clients can be quick to request creation of new VMs, snapshots, and addition of space when there is an immediate need before checking or understanding what other VMs resources may be impacted. Another example of improper planning is setting up HA and not adjusting settings after one or more VMs have been recently created.

haripala
Level 11

HI vladan

Please let me know the perfect virtualization management tool...

techbender
Level 10

The issue that we seem to encounter is the other departments that we work with will put in a request for us to spin up a VM using specs that a 3rd party says they need. We create the VM and after awhile when they experience an issue they say they need more resources, usually ram. There's been a couple instances involving a SQL database where they would look at the ram being used and see that the database had taken up all of it and immediately attributed these performance issues to it not having enough ram. Those of you familiar with SQL will know that by default the it will try to use as much ram as you give it. Those instances where they had problems it was actually just that VMware tools needed to be installed. Just goes to show you gotta trust but always verify that what they're telling you is correct instead of just throwing more and more resources at it. Good info about defining rules for VM's, We keep size quota's on mailboxes and user's share drives. Makes sense that would apply to VM's also.

vladan
Level 9

If there is one, everyone would new... No, there is not a single perfect virtualization management tool. You always have to compare the features you can and when in testing phase double check with CLI tools like ESXTOP in VMware vSphere. The testing full featured tool during 30 days is always a challenge, especially as if an enterprise admin you have thousands other daily tasks to do, but it's something which should be put on the TO-DO list as high as possible. As good management tool can save you time and perhaps sometimes also your job! -Smiley Happy.

dwoj
Level 9

"VMs with multiple vCPU when only single vCPU needed - start with less to add more later."

Outside of technical context, this is good advice for life.  (/end random thought)

jbakervt
Level 8

"check the requirements and follow the documentation (RTFM)"

That says it all... if you know what you really need, you won't provision resources incorrectly. If you don't RTFM, you will not know how to do it correctly.

"VM sprawl"

I predict this will be the newest over-used metaphor in sales pitches. However, it is an apt metaphor, as thoughtless development leads to poor use of resources.

hulattp
Level 9

Liking some of the tidbits of info, thanks everyone.ibublak

byrona
Level 21

I completely agree that you need good tools to manage a virtual environment; I really don't think you can be very successful without them.

On the flip side, I also need to agree with Jfrazier in that the tools themselves can suffer from many of the problems as your virtual environment; tool sprawl, different teams using different tools; tools not being properly maintained (bad data in = bad data out), etc.  Take care of your tools and they will take care of you! 

esther
Level 12

Great Topic vladan, I like your phrase byrona ... "Take care of your tools and they will take care of you"

zackm
Level 15

Jfrazier hit the nail on the head.

You can spend all the money in the world adding tools to tell you when things are a little off, but if you don't know and/or understand the underlying technology, you can't really expect the tool(s) to give you a root-cause analysis. To be fair, the tools are getting closer and closer to this, but the need for reading the manual is something that should never be ignored!

mikegrocket
Level 10

A virtual envoironment is no different than anything else. Discipline must be exercised or else you windup with sprawl. Being lazy about maintaining your networks, right sizing your VM's with cpu and memory, etc, these are all important considerations. And there is no "silver bullet" in you tool holster. You must test and decide which one works best for you.

weiss
Level 9

I am new to virtualization. These posts are very informative. Thanks!

automag928
Level 12

Virtualization is starting to become a lot more prevalent with the things my company has been doing lately.  I've recently had a similar discussion with my co-workers and others and these points brought up are indeed good ones to be aware of.  Thanks for the great post!

prowessa
Level 12

That is very nice.

prowessa
Level 12

vladan Very nice post this will really help us all.

Thank you.

kurtrh
Level 13

Good tools to manage the virtual environments is definitely needed. Without the right tools, and your understanding of said tools, you Virtual environment could suffer. But on another point make sure the tools are installed on a server that you know will not go down for any reason. We try to place those tools on hardware based servers, rather then in the same virtual environment that we are trying to monitor for problems. If you have your tools on that virtual environment and it starts having problems, you do not have your tools anymore to figure out what the problem is.

zackm
Level 15

This seems intuitive, but is actually a great point. I have seen a LOT of environments hosting their tools within their virtual environment. Sometimes even on the same host as their critical apps!

prowessa
Level 12

Thank you for this information.

esther
Level 12

prowessa.... he does have a good point. Thanks Kurt H

blsanner
Level 12

Many good points being made in this thread.  I agree with what most others have said: Having a good virtualization monitoring solution is a great benefit.  But, it's a requirement to understand the underlying infrastructure and being able to correlate what the tool is telling you to actionable info.

sqlrockstar
Level 17

As a DBA my answer is "Yes, I really do need 16GB of RAM".

I've been preaching for years that bad configuration choices are a leading cause for performance issues. Nice to see someone else singing the same tune!

eno
Level 10

Yes Vladan, i completely agree with you. No one product can do it all and most of have a bunch of work to do that we do not have enough time to test out products to the fullest.

jkump
Level 15

Good information.  Keeps the knowledge-base growing. 

About the Author
Virtualization blogger and IT engineer, living at Reunion Island (fr). Trying to help others with their journey to all virtual...