A VMKernel is a critical component of any virtual environment. VMKernel is the reason why your VMs are allocated with memory, CPU, and other resources from the physical hardware. What makes the VMKernel special is that it runs directly runs on the ESX/ESXi host. However, this doesn’t mean that performance and latency issues are particularly your VMs. They can be also be found in the host itself, which can cause sudden I/O spikes affecting the VM’s performance.
If you think your VM performance is poor due to I/O spikes, you need to go to the additional step and drill into that host to look at the kernel I/O latency metric. This metric will show whether or not the host has I/O spikes. Experts recommend that this value should be around 0 to 1 millisecond. If you see values over 1 millisecond, then your VMs may not perform the way you want them to.
In addition to monitoring the kernel I/O latency, you should also monitor the queue latency counter. This measures the average amount of time the data is in the queue. There should be absolutely no data in the queue and the value of this counter should be nothing greater than 0. If the value is greater than 0, then it means workload in the VM is very high and data cannot be processed with high efficiency, which will lead to I/O spikes and performance issues.
In a virtual environment, problems start slowly and work their way up affecting other hosts in a group, VMs in a host, OS, and resources. Other measures can be taken to keep the host I/O latency to a minimum. For example:
- One way to reduce I/O spikes is to increase virtual memory. For this to work, you should consider increasing the host memory. This way the system memory is utilized to store data thereby avoiding disk access.
- Determine how your storage arrays are performing. When there are too many VMs trying access the storage system, a bottleneck occurs.
- Look at ways to balance disk loads across your disk drives. This way efficiency improves and there’s nothing stuck in queue.
In order to proactively troubleshoot virtual host I/O latency issues, you need a virtualization management software that will monitor all your virtual hosts, giving you key insights into critical performance metrics. You can never go wrong with virtualization management software because it allows you to do the following in real-time:
- Identify and troubleshoot the root cause of kernel I/O latency in the VM host. See how each of your ESX/ESXi hosts are performing – drill down to see any abnormal behavior in each host.
- Ensure the VM has enough resources to perform smoothly.
- Get a wealth of critical information about your storage performance through an intuitive and customizable dashboard.
Virtualization bottlenecks can be tedious to troubleshoot if you don’t know where to start looking. Watch this video where vExpert Scott Lowe shows you how to monitor kernel I/O latency using virtualization management software.