Skip navigation
1 2 Previous Next

Geek Speak

17 Posts authored by: Mike Thompson

On Tuesday, February 24, we released new versions of all our core systems management products, including Server & Application Monitor, Virtualization Manager and Web Performance Monitor. We also released a brand new product called Storage Resource Monitor. While this is all exciting in and of itself, what we’re most thrilled with is that all these products now have out-of-the-box integration with our Orion platform and include our application stack (AppStack) dashboard view.

 

The AppStack dashboard view is designed to streamline the slow and painful process of troubleshooting application problems across technology domains and reduce it from hours—and sometimes even days—down to seconds. It does this by providing a logical mapping and status between the application and its underlying infrastructure that is generated and updated automatically as relationships change. This provides a quick, visual way to monitor and troubleshoot application performance from a single dashboard covering the application, servers, virtualization, storage, and end-user experience. What’s key here is that this is all done in the context of the application. This means that from anywhere in the AppStack dashboard, you can understand which application(s) is dependent on that resource.

 

In addition to the immediate value we think this will have for you, our users, it also highlights a shift within SolarWinds towards setting our sights on tackling the bigger problems you struggle with on a daily basis. We’ve always sought to do this for specific situations, such as network performance management, application monitoring, IP address management, storage monitoring, etc., but the new AppStack-enabled products help solve a problem that spans across products and technical domains.

 

However, this doesn’t mean SolarWinds is going to start selling complex, bloated solutions that take forever to deploy and are impossible to use. Rather, by intelligently leveraging data our products already have, we can attack the cross-domain troubleshooting problem in the SolarWinds way—easy to use, easy to deploy and something that solves a real problem.

 

But know that the AppStack dashboard view isn’t the end. Really, it’s just the beginning; step one towards making it easier for you to ensure that applications are performing well. Our goal is to be the best in the industry at helping you answer the questions:

 

  • “Why is my app slow?”
  • “How do I fix it?”
  • “How do I prevent it from happening again?”

 

While integrating these four products with the AppStack dashboard view is a great first step, there’s clearly a lot more we can do to reach the goal of being the best in the industry at that. Pulling in other elements of our product portfolio that address network, log and event, and other needs—along with adding more hybrid/cloud visibility and new capabilities to the existing portfolio are all areas we are considering to reach that goal.

 

Easy to use yet powerful products at an affordable price. No vendor lock-in. End-to-end visibility and integration that doesn’t require a plane-load of consultants to get up and running. That’s our vision.

I hope you’ll take a look for yourself and see just how powerful this can be. Check out this video on the new AppStack dashboard view and read more here.

A Guide to Navigating Around the Deadly Icebergs that Threaten Virtualized Databases


When it comes to virtualized workloads, databases can be in a size class all to themselves. This and other factors can lead to several unique challenges that on the surface might not seem all that significant, but in reality can quickly sink a project or implementation if not given proper consideration. If you're put in charge of navigating such a virtual ocean liner-sized workload through these iceberg-infested waters, you'll want to make sure you're captain of the Queen Mary and not the Titanic. How do you do that? The first step is understanding where the icebergs are.I recently had a conversation with Thomas LaRock, president of the Professional Association for SQL Server (PASS) who also happens to be one of our Head Geeks here at SolarWinds, to get his input on this very topic. Here's what I learned:

 

CPU & Memory Allocation

First, don't treat a database like a file server when it comes to configuration and virtual resource allocation. Typical configurations allow over allocation of both memory and CPU. However, configuration of CPU shouldn't be more than 1.5-2 times the number of logical cores you have. When it comes to memory, don't over allocate at all if possible, instead going to at most 80 percent. As memory utilization gets near 100 percent, you may not have enough resources to even reboot the host. If you do push your systems, make sure that you not only have a virtualization monitoring tool, but that you actively use it.

 

High Availability Strategy

Most virtualization admins use snapshots and vMotion-Thomas' preferred option-as a primary approach to address high availability concerns. On the Windows side specifically, clustering and availability groups are also common. While either technology can be effective, they probably shouldn't be used together. An example of why not is a database VM being vMotioned to another VM as a result of a performance problem, but the availability group just seeing that as a server instance no longer responding. If you do use both, make sure that you don't allow automatic vMotion so there is always an operator in the mix to (hopefully) prevent problems, otherwise bad things can happen.

 

To Monster VM or Not?

You might wonder if an easy way to overcome the challenges of virtualized databases is simply to allocate one of VMware or Hyper-V's "monster VMs" to the database instance and just solve problems by throwing hardware at them. However, a better approach is to put database VMs on a mixed use host that includes a range of production, development, test and other workload types. The rationale being that if you have a host with nothing but one or more database servers running on it, you have no options if you accidently run out of resources. With a typical mixed use host, you're less likely to be simultaneously hammering one resource type, and if you do start to hit a resource bottleneck, the impact of turning off a development or test VM to provide short term resources will typically be less than shutting down a production database.

 

Taking these considerations and tips into account can help make sure your virtualized databases stay afloat a long time rather than being lost due to a potentially avoidable iceberg on the maiden voyage.

If you're looking for additional information on virtualizing databases, SolarWinds has a number of great white papers available online, including "5 Risks for Databases on VMware" and "Monitoring Database Performance on VMware."

 

Note: This post originally appears in VMBlog at http://vmblog.com/archive/2014/10/23/would-you-rather-captain-the-queen-mary-or-the-titanic.aspx#.VGUUxfnF8uK

 

In the VMworld 2013 keynote session in San Francisco Monday morning VMware made a number of announcements mostly related to software-defined data center (SDDC) capabilities.  While they focused a lot on networking, storage and hybrid cloud, the capability that will probably have the most impact in the short term will be the increased horsepower of vSphere 5.5.  As CEO Pat Gelsinger described it, the latest version of vSphere is “2X” the horsepower of the previous version with double the cores, double the number of VMs and more than double the memory.  VMware was clear about why they think this is important, they want to take away any barriers that prevent people from moving mission critical applications to the virtual environment.  While this will help accelerate the movement of these applications, this additional VM capacity is only one part of the story.  Storage and auxiliary capabilities like replication and high availability (HA) are also required to make many companies comfortable with virtualizing applications like their mission critical databases.

 

Fortunately, on the storage side there has been a lot of focus on addressing the storage I/O issues that have long been a limitation for virtualization.  Various solid state disk (SSD) technologies along with capabilities like what VMware, Microsoft and others are driving for software-defined storage are helping improve the storage side of things.

There are a number of sessions at VMworld addressing the other required capabilities like HA including specific sessions on SQL databases, virtualizing Exchange, Hadoop and SAP. This is being attacked from both the application vendors and VMware with at least the basic tools and a set of procedures that can be used to do things like perform a rolling software update with relatively low disruption to the application availability.

 

However, the impression across the board is that while it is possible to run and manager many of these mission critical apps from a compute, storage and HA point of view, it still isn’t easy or mainstream yet.  Many of the approaches discussed require a high degree of knowledge, fairly complicated procedures, higher end and more expensive hardware and software, and a degree of compatibility across vendors that isn’t always there.  As a result, this is still somewhat of a leading edge technology for many companies.  There is clearly enough interest and demand in the market to continue to drive rapid improvement and maturity across the board.  In the short term, many of these gaps can be patched over by skilled IT admins if given the right tools. 

 

While there are things needed to mature the mission critical application space from companies like VMware and the storage vendors, one of the simplest ways to start is by getting end-to-end visibility across the application stack.  The visibility provided by extending virtualization management to apps and storage arms the skilled admins with the information they need to intelligently fill in the gaps that still exist between application, virtualization and storage management to ensure that the end business service is not impacted.  That is exactly the feedback we got from the SolarWinds customer base when developing integration between the SolarWinds Virtualization Manager and Server and Application Monitor (SAM) products on top of existing Storage Manager integration. The integration provides full visibility across the virtualized application stack from the application and physical servers to the virtual environment all the way down to the storage LUNs and arrays.  In additions, SAM’s upcoming release will have some critical capabilities around database monitoring that will help complete the mission critical visibility story.

 

The net result, the broader ecosystem is moving closer to making virtualized mission critical applications mainstream and easier but end-to-end visibility is here today.

Virtualization and automation have become mainstays of the modern data center with the latest versions of vSphere® and, more recently, Hyper-V®, as well as increasing automation built into applications, storage arrays and networking. These technology innovations have greatly increased the speed and flexibility with which changes can occur in each of these areas. Today, IT professionals can create new virtual machines in minutes and move existing machines across town without downtime.

 

The problem is that this innovation and flexibility has developed largely within silos, such as those created by compute, storage and applications. While the ability to make changes within an individual technology area rapidly and automatically has evolved, the ability to coordinate with the other technology areas affected by those changes has not kept up. For example, if one VM is overloading the CPU of a given host, the hypervisor can move the VM easily and automatically to a host that has more CPU capacity. However, that move may not be optimal for the underlying storage systems, so that fixing the CPU problem creates a storage bottleneck.

 

What is needed to address this problem? Many groups and companies are working on it, with capabilities such as software-defined data centers or cloud orchestration aimed at improving the cross-domain automation and coordination.

 

Unfortunately, having one company’s technology control and manage (and some would say commoditize) all the supporting infrastructure of the datacenter is not in the best interest of many of the big IT players, or for that matter, the consumer. As a result, cross-silo automation has not yet materialized, which leaves the responsibility for much of that cross-domain coordination with the administrators running the systems. That also increases the need for cross-domain visibility and monitoring so that administrators can see across domains and can get the information they need to coordinate between the various infrastructure and application environments in the data center quickly and automatically.

 

That is where SolarWinds® comes in. With SolarWinds Virtualization Manager Version 6.0 releasing today, we have begun putting together the end-to-end visibility that is becoming more and more necessary to keep up with the rapid change occurring in the individual technology silos. Virtualization Manager 6.0 includes integration with Server & Application Monitor (SAM) to provide visibility across the full application stack from Applications to VMs, hosts and datastores to physical server hardware and even all the way down to the storage disks via integration with Storage Manager (STM).

 

In a recent survey of our systems admin customer base, 47% of respondents indicated that the hardest problem for them to answer was, “Is the problem with my application being caused by virtualization or storage?” Additionally, having a “Single View of Application, Virtualization and Storage Data” ranked as the most important capability needed by the survey respondents.

 

With this latest release, a customer can now see the key components of the application’s underlying infrastructure in context of that application. Alternatively, the virtualization administrator can quickly understand all of the applications that are running on a given resource, such as a host or datastore, and then take appropriate actions to optimize resources and performance.

 

By providing that integrated, single view of the end-to-end application stack from within the SAM console with Virtualization Manager, administrators can now better coordinate across the silos. In this way, they can do more than optimize one resource area—they can optimize for the application (i.e., the business service) that customers need.

  

Specifically, the new capabilities in Virtualization Manager 6.0 include:

 

  • Integration with Server & Application Monitor (SAM) for in-context views of the virtualized application stack from application to virtualization to the datastore/storage
  • Deeper virtualization visibility to the datastore for SAM & NPM users
  • Enhancements to Microsoft® Hyper-V storage objects (e.g., clustered storage volumes)
  • Updated libraries and components, and improvements to the GUI, collection speeds and supportability

 

The integration will be delivered with Virtualization Manager 6.0 and will be part of SAM, so implementation should be simple. Combined with the very affordable price of both SAM and Virtualization Manager, end-to-end visibility should be very accessible to all types of users. At SolarWinds, we are still trying to keep it simple, but that doesn’t mean we aren’t taking aim at the big problems!

Josef Cilia of APS Bank in Malta kindly shared his experience using SolarWinds Storage Manager (STM) with us.   We got some great insight into how STM is helping Josef keep his systems up and running and be more proactive in managing his systems with visibility all the way from servers and fabric to his arrays.  Josef’s feedback is provided below.


JC: Since I’ve installed Storage Manager, I can say that I have better piece of mind regarding my storage environment. Storage Manager not only helps me tackle problematic issues on my storage but also provides me with forecasts/projections for my storage (it tells you when it will reach 80%, 90% and 100%), immediate response by email whenever there are problems plus I can also monitor the status of serves such as the HBAs, memory, CPUs, disks and network.


Basically I’m monitoring 3 enterprise storages arrays namely a HP EVA4400, a HP EVA4100 and an IBM DS3524 (85TB of RAW data in all) together with 6 Brocade switches and 21 servers (up until now!).

 

SolarWinds: Since you implemented STM, what insight has it provided to you?


JC: Among the many helpful things that I have gained from installing this product is that I can drill down on every single LUN on my storage arrays and view detailed information such as total IOs/sec, read/write IOs, MB written/sec. and read/write latencies. The same data is also available for disks and disk groups. All data is displayed in graphical format.


Another simple but important thing is that I can check the redundancy on my controllers. I’ve never had such a visible picture where I can view the load on my storage controllers. It is helpful when it comes to viewing the top 10 LUNs by total IOPs, top 10 LUNs by total latency, top 10 LUNs by reads, top 10 LUNs by writes. These can be viewed at a glance. On my storage, I set rules to alert me whenever there are read access delays, write access delays, disk queue lengths, and when threshold usage is greater than 95%.


With regards to my fabric, now that I installed Storage Manager I can monitor the switch as a whole or individually port by port. This data is shown in a single screen where I can view the status of the ports (online/offline) or all the zoning on that particular switch. This tool also monitors the board temperature, fans status, power supplies status. I also set rules on these switches in such a way that if a port goes offline it alerts me immediately by sending an email. I have also set another threshold to alert me when there are errors/sec. generated on ports.


I’m also monitoring most of my servers connected to my fabric. Although this tool gives detailed information on the performance on the CPUs, disks, memory and network (which is more helpful to the server administrator), I manage the files stored on my storage meaning I can view which files are large and are using my storage, which files are orphaned and can be removed from the storage, duplicate files and what type of files are stored on my storage such as MP3s, images, DB files and many others. I run scheduled reports to extract such information from my storage and present these reports to my management on a monthly basis. I set some notifications on servers to alert me such as whenever there is high CPU or memory usage.  Another useful thing about Storage Manager is that servers do not require any restart after installing or uninstalling STM agent on them.


I use STM reporting for monthly progress reports which I present to the management. These reports are very easy to extract and they can be extracted in no time where in the past I had to spend about 3 hours to issue those reports. Another useful feature about this product is the way notifications are configured. Nowadays, I’m not wasting my time fire-fighting problems but now I’m able to be more proactive.


SolarWinds: How many issues have you found proactively that would have otherwise ended in a service performance issue or outage?

 

JC: Two within a month.

 

SolarWinds: Would you recommend SolarWinds Storage Manager and if so, why?


JC: Definitely yes. STM is helping me to forecast my storage, be more proactive before problems arise, have better visibility of all my SAN, have a visibility of the performance of my storage in a single screenshot, and much more.

VDI Storage

VDI workloads have made virtual and storage admins look for storage technologies that have higher I/O rates and better performance. While solid state drives (SSD) can be a viable solution to increase storage performance as a whole, administrators still worry about TCO and ROI over the long run. This can ultimately lead to rejection of SSD and selection of cheaper mechanical disks that may not meet the VDI performance requirements and cause bottlenecks.


Matching physical desktop IOPS with mechanical disk is one of the biggest roadblocks in VDI deployment. Let us compare the IOPS performance and cost for different types of hard disk.


Characteristics

Enterprise Mainstream

Performance SSD

(TOSHIBA MK4001GRZB 400GB)

Enterprise

Performance SSD

(Kingston SSDNow E100 400GB)

10K SAS Drive

(Western Digital Velocity Raptor)

Drive Interface    

6Gb/s SAS

SATA/300

6Gb/s SAS

Maximum Read Transfer Rate

510 MB/s

535 MB/s

200 MB/s

Write Transfer Rate               

230 MB/s

500 MB/s

200 MB/s

Performance

Read - 90000 IOPS

Write –17000 IOPS

Read - 52000 IOPS

Write – 37000 IOPS

 

Target Storage and Server

Mid-High Range

Low-Mid Range

 

Price

$7,148.35

$1,399.99

$245

 

















Price-GB.jpg


4K Read.png

Storage Read IOPS/Queue Depth

 

4K Write.png


Storage Write IOPS/Queue Depth


The Price/GB comparison shows us the price of a mainstream SSD to be approximately 10 times the price of a mechanical hard disk. But the catch here is the read/write IOPS for SSD is 20 folds more than a mechanical hard disk. Hence using the right storage type for the right purpose is the key for a good storage investment.


For instance let us assume your enterprise has 100 clients on a host, and the hard disks are on RAID1 configurations.

  • For Windows 7 clients to run smoothly the read/write ratio is 40/60 and the IOPS needed is 10 per VM
  • Based on this for 100 VMs we need 1000 IOPS of which 400 is used for read and 600 for write.

 

Storage Cost Calculation (10K SAS drive in RAID1 vs. Enterprise Mainstream Performance SSD_


For RAID1 the IOPS is 50-60 for read, and 150-160 for write. That implies:


  • (400/60) + (600/160) = 10.41 so that’s approx. 11 disks for 100 VMs
  • 11 (disks) * 245 (price of 10K SAS) = $ 2,675 for hosting 100 VMs on 4,400 GB of storage

On the other hand, a single enterprise SSD can handle all the IOPS needed by the 100 VMs, costing about 3000USD having an IOPS rate of 90k for read, and 17k for write operations. But, on the flipside, you are left with 400GB of storage which is just 10% of what 10K SAS offers.  For a larger environment that would use all more of the SSD IOPs capacity, a combination of SSD and SAS drive can provide the most cost effective combination of IOPs and storage space.  By itself this makes SSDs more expensive, especially if you are looking for storage space and NOT performance. From there it is fairly simple math to calculate TCO and ROI for each hard disk type once you know your VDI infrastructure, the number of VMs and VDI usage and possibly the current IOPs identified with virtualization management software.


Storage administration in VDI is all about managing storage capacity and IOPS demand. Balancing this with keeping cost in mind is key for TCO and ROI. Otherwise, the whole purpose of deploying VDI to save IT cost may tend to fail.


A few points on how to maximize your storage performance and ROI are provided below.


  • Use the Right Storage Device: To address this capacity vs. IOPS issue we can adapt the process of adding SSDs to handle I/O operations while using mechanical disks for back-end capacity
  • Thin Provisioning to Save Storage: Thin provisioning of VMs on faster storage can save space on SSD hard disk. This allows more VMs per disk (but remember to monitor storage capacity as exceeding physical capacity can shut down your VMs).
  • Memory Swapping: Having memory swapping occur on a SSD will allow applications that use swap memory to run faster while the actual processing is done on mechanical hard disks
  • Segregate Based on Usage: You can segregate the usage of VM based on usage or departments. For example, for a department developing and testing VMs requiring faster I/O can run on a SSD while the marketing and sales that needs less IOPs performance can run on mechanical disk
  • Moving the Snapshots: Non-active snapshots can be moved to a backup mechanical hard disk so it provides more average I/O for active snapshots
  • Storage Tiering: Storage tiering is the process where the software determines the data based on type, usage and provisions it on mechanical or SSD disk automatically.
  • Boot Storm Management: Boot storm occurs when all virtual desktops are turned on at the same time causing a spike on storage I/O usage. Timed boots or aligning storage arrays based on power on time can avoid boot storms.

In the busy world that most IT admins live in, it can be pretty hard to carve out even twenty or thirty minutes to watch a training video. So when we sat down with some of our favorite bloggers to put together a couple comprehensive training videos that were a bit longer, we made sure we could also have a set of short form videos on very targeted topics.

 

We have a set of seven initial videos that I call our “Two-Minute Tutor” video series. Not being known for subtle naming, you probably already guessed that these videos are all about two minutes long and show you how to evaluate or troubleshoot common virtualization or storage issues.

 

The first two I’d like to highlight are focused on storage I/O analysis, which really go hand in hand. The first video “Storage I/O Latency Impact Analysis” focuses on management of a VMware datastore with high storage I/O latency and drilling down on that datastore to gather more details. However the focus of this video is how you can map that datastore to hosts, clusters, and VMs that are dependent on it. Once you have the environment map, it is easy to find related resources that are either causing the problem or being affected by the problem. Check out the video here:

 

Virtual Storage I/O Latency Impact Analysis



The second storage I/O latency video is “Storage I/O Throughput Analysis.” Given that you’ve mapped and understand the effects of a virtual storage I/O latency problem, this video walks you through the steps to identify the drivers of storage I/O for a datastore. After drilling down on the datastore, the video shows overlaying the VMs hitting that datastore so that you can see which VMs are driving storage I/O, spikes in I/O, etc.

 

Virtual Storage I/O Throughput Analysis

 

 

 

Other Two-Minute Tutor videos include:


 

We would appreciate feedback on the videos and the usefulness of their short format. We’d also like any input on other virtualization or storage topics you think would be useful.

It’s been 35 years since the very first solid-state drive (SSD) was launched. This went by the name “solid-state disk”. These were called “solid-state” because they contained no moving parts and only had memory chips. This storage medium was not magnetic or optical, but they were solid state semiconductors such as battery-backed RAM, RRAM, PRAM or other electrically erasable RAM-like non-volatile memory chips.

 

In terms of benefits, SSDs worked faster than a traditional hard drive could in data storage and retrieval – but this came at a steep cost. It’s been a constant quest in the industry, over the years, to make the technology of SSD cheaper, smaller, and faster in operation, with higher storage capacity.

A post in StorageSearch.com shows the development and transformation of SSD, over the years, since its first usage and availability until now.


Why Storage and Datacenter Admins Should Care?


More than the user using the PC or notebook, it’s the storage admins who are spending time managing and troubleshooting the drives for detecting storage hotspots and other performance issues. And it’s imperative that storage and datacenter admins understand the SSD technology so they can better apply and leverage the technology in managing the datacenter.


Application #1 – Boosting Cache to Improve Array I/O


A cache is a temporary storage placed in front of the primary storage device to make the storage I/O operations faster and transparent. When an application or process tries to access the data stored in the cache, this can be read and written much quicker than from the primary storage device which could be a lot slower.

 

All modern arrays have a built-in cache, but SSD can be leveraged to “expand” this cache, thus speeding up all I/O requests to the array.  Although this approach has no way to distinguish between critical and non-critical I/O, it has the advantage of improving performance for all applications using the array.


Application #2 – Tiering – Improving Performance at the Pool Level


SSDs help storage arrays in storage tiering by dynamically moving data between different disks and RAID levels in meeting different space, performance and cost requirements. Tiering enables storage to pool (RAID group) across different speeds of storage drives (SSD, FC, SATA) and then uses analysis to put frequently accessed data on the SSD, less frequently on FC, and least frequently on SATA.  The array is constantly analyzing usage, and adjusts how much data is on each tier. SSD arrays are used in applications that demand increased performance with high I/O. It is often the top tier in an automated storage tiering approach. Tiering is now available in most arrays because of SSD.


Application #3 – Fast Storage for High I/O Applications


Since arrays general treat SSD just like traditional HDD, if you have a specific high I/O performance need, you can use SSD to create a RAID group or Storage Pool.  From an OS and application perspective, the operating system understands the RAID as just one large disk whereas since the I/O operations are spread out over multiple SSDs, thereby enhancing the overall speed of the read/write process.

 

Without moving parts, SSDs contribute towards reduced access time, lowered operating temperature and enhanced I/O speed. It should be kept in mind that SSDs cannot handle huge data, and data for caching should be chosen selectively based on what data will require faster access to – based on performance requirement, frequency of use and level of protection.


What’s Best in Virtualization?


In a virtualization environment, there is the fundamental problem of high latency with host swapping primarily in traditional disks compared to memory. It takes only nanoseconds for data retrieval from memory whereas it takes milliseconds to fetch from a hard drive.

When SSDs are used for swapping to host cache, performance impact of VM kernel swapping reduces considerably. When the hypervisor needs to swap memory pages to disk, it will swap to the .vswp files on the SSD drive. Using SSDs to host ESX swap files can eliminate network latency and help in optimizing VM performance.


Conclusion


The application of Solid-state Drives has become significant in achieving high storage I/O performance. Proper usage of SSDs in your storage environment, along with the right set of SAN management and performance monitoring tools and techniques, can ensure your data center meets the demands of today’s challenging environments.

 

If you are interested to learn more about the advantages of SSD over Hard Disk Drives (HDD), you can look at this comparative post from StorageReview.com.

The term virtualization is synonymous with being a technology that is easy, cheap and effective, amazingly packaged in one technology. But because many of the normal limitations found in deploying physical servers have been removed, it can get out of control and begin causing its own problems.


VM creation is quick and easy, and because it does not come with the extra hardware baggage, system administrators are more than happy to create new virtual machines whenever needed. But once the job is done, these VMs tend to be ignored and, because they are residing on your physical infrastructure somewhere, they tend to consume essential storage and processing resources leading to VM Sprawl.

 


Stray VMs, a Bane to Admins


Stray VMs are not something that can be avoided, but the problem gets more complicated as additional VMs are added and the admins shift their attention to the management of these individual systems and away from the overall health of the environment allowing the resources consumed by VM sprawl to get bigger.

All the active VMs tend to be attended to, but the old unused ones previously created, still consume physical and financial resources, and even when powered off, these stray VMs may still have software licensing and support costs attached that can prove to be an organizational burden. Not just that, if the unused VMs are over allocated with memory, storage and processor, then there is a tremendous waste of valuable resources that can be channeled elsewhere. Quickly summarizing, virtualization administrators should keep an eye out for the problems that VM sprawl can cause.

Idle but alive

  • Zombie VMs may seem idle but many may still consume CPU and storage resources, burdening hypervisor performance and wasting expensive storage capacity.

Resource over allocation

  • Because of the ease of creating VMs, admins sometimes end up creating VM’s and over-allocating resources.  By right-sizing these VMs closer to their actual necessary levels, it can be a cost-effective way to free up more memory and storage.

Storage resource crunch

  • Unused VMs occupy storage resources which can be used elsewhere in a virtual environment.  In addition to this, VM snapshots also consume huge storage resources and multiply the VM sprawl impacts.

Software Licenses

  • Along with consuming critical resources, VM sprawl also can use software licenses which may lead to software violation or at a minimum can make it very difficult to get an accurate license count.

Capacity Planning

  • Due to the increased number of stray VMs the administrators may plan future virtual expansion based on the wrong assumptions


VM security tips

VM Sprawl can also represent a significant security risk as VMs that are offline may escape routine security processes. Some of the following best practices can be used as quick tips for effective management of security issues related to VM sprawl security. Have a process to ensure all virtual machines, including those offline, are fully patched and secured on an ongoing basis or at least before they are brought back into the IT environment.

  • Appreciate the architectural differences of a virtual environment including VM sprawl issues and adapt security policies accordingly
  • Apply intrusion detection and antivirus software to all physical and virtual layers
  • Avoid VM sprawl, enforce policies to ensure VM creation is closely monitored and machines are decommissioned after use.
  • Use secure templates for the creation of new VMs

 

VM Sprawl Control Super Hero


 

Controlling this situation with the right set of virtualization management tools is essential for the over-worked admins to  put the brakes on virtual machine sprawl by quickly having a handy, affordable and easy-to-use control mechanism that can start governing the pesky situation of VM sprawl quickly and easily can give the admins one less thing to worry about.

Organizations with large data centers often can make storage management difficult by using more than one kind of storage. This type multi-vendor storage management has made it complex for administrators to manage storage devices.

In a heterogeneous storage environment administrators struggle with:

  • Enterprise capacity Planning: Administrators need to perform an analysis on the RAW capacity, storage pool usage, RAID capacity, free LUN’s etc. in order to forecast and allocate storage resources. This ensures all LUN’s are allocated and thereby reduces waste
  • Mapping VM’s: Administrators need to map their VMs to the underlying LUN’s associated with the VM’s storage. This helps identifying unused VM’s and associated storage resources
  • Storage Bottlenecks: Storage read/write speed determines the usability of certain applications. Any critical glitch with any storage components may affect the entire storage networks performance. It is important to maintain an optimum read/write speed and determine if any bottlenecks exist.
  • Updating Storage Components: For devices on the network, administrators need to know the asset information about the storage components such as serial numbers, firmware versions, etc. for compliance and audit reporting.

 

IT administrators need to monitor and report report on each of these aspects of their storage environment across all vendors. By having all of the storage information in one place they can correlate errors and identify performance issue root causes related to all storage components. Managing an array of storage devices, monitoring performance issues across all vendors can be quite a task for storage administrators.

Best Practices involved in multi-vendor storage management

CPU Load will affect Storage Performance – CPU load of storage servers will affect the storage performance, ensure CPU load does not exceed the optimum level. Setting an SNMP trap to trigger alert will allow you to know when the CPU reaches its threshold limit.

Disk Load will affect Latency – This can be tracked by aggregating all disk read/write speed in a server. When there is latency it will cause slow read/write speeds from which you can pinpoint the volume that is overloaded. If a volume is overload, you can identify additional capacity to allocate more disk space or to migrate volumes.

Effective Storage Capacity Planning – Plan your storage needs ahead of the requirement, keep at least ¼ of the total storage free and don’t let the storage reach it maximum level. Always have a storage buffer available that allows you to move or backup VM’s and files when needed. Identify all stale VM’s and unused LUN’s for proper utilization of storage.

Defragmenting Volumes – Defragmenting may sound as a simple process but the process may slow down the servers.  As a result, plan your maintenance schedules ahead of time, so it does not affect productivity. The best practice is schedule storage maintenance once in a month to remove clutters and fragments in storage devices.

Storage Tiering – It is a process to put the data where it belongs storing non-critical data like snapshots and stale VM’s to low speed storage devices and alternatively data highly used to the high speed disks. This kind of prioritizing storage need will reduce cluster and optimizes storage performance.

Reports and Notification – Keep a note of all asset information. This helps you to update firmware or to find out the warranty, EOL status of components.

In a diverse storage environment, multi-vendor monitoring and reporting is essential to keep track of all storage related performance issues. Storage is one of the major layers in IT network, so any performance drag may affect productivity, end-user usability or application availability, so be pro-active and keep your storage spindles spinning.

 

When he’s not blogging, Michael McNamara is a technical consultant for a large health care provider in the Philadelphia area and a husband and father to three girls. He describes himself as a “jack of all trades” in his current job, with primary responsibility for data and voice infrastructure while also supporting Windows Active Directory, Microsoft Exchange, Microsoft Windows, VMware, FC SAN, and more.

 

URL: http://blog.michaelfmcnamara.com/

Twitter: @mfMcNamara

 

MT: How did you start blogging?


MM: I started blogging in 2007 in an effort to build my digital persona when my job was threatened by a potential sale of the department I worked in. It was my initial goal to be the #1 ranking for “Michael McNamara". The idea being that when I interviewed and people Google’d “Michael McNamara” they’d quickly see that I was legit and the real deal. The job scare blew over but I had found a new hobby in blogging.

 

MT: As your blog has grown, who are your readers and what are they looking for from your blog?


MM: At the start I kept my posts geared towards Avaya (formerly Nortel) equipment, specifically Ethernet switching and IP telephony since there was a real lack of information around Avaya equipment available on the net.. There was little useful documentation, no community, and only pre-sales spec sheets that weren’t much use to the engineers and system administrators actually working with the equipment. As I solved my pain points and shared them online I found that I wasn’t alone. There were people all over the world struggling with the same problems and the blog kind of caught fire.

 

While I actually work with and support numerous technologies I originally limited my blogging to topics which were absent from the net. I felt there was no benefit in me writing about topics that were already covered in-depth elsewhere on the net. A few years ago I realized that people valued my feedback and experiences so I started expanding my topics beyond just Avaya networking.

 

MT: What new trends are you seeing in your interaction with your readers?


MM: About two years ago I started a discussion forum to provide a place for guests, strangers even to ask their own questions. The comment threads on my blog posts were becoming inundated with off-topic comments and questions and I didn’t want to turn people away just because their question wasn’t related to the topic of the blog post so I decided to start a discussion forum. On the discussion forum users could ask whatever question they wanted and myself and a few other subject matter experts would try to answer them as best we could. With more and more projects being “value engineered” there are a lot of system administrators and engineers trying to deploy and configure the equipment themselves, hence there are a lot of questions and advice being sought.

 

MT: Any new technology trends you are hearing from your community?


MM: There are a number of hot buzz words in the industry today including OpenFlow, SDN, BYOD, etc. One of the areas I’m working with in healthcare is access to clinical information. While some of that potentially involves BYOD there are also technologies available from Citrix and VMware that allow physicians access to traditionally fat applications from smartphones and tables. With more and more emphasis on ‘value engineering’ and change management I’m also looking at automation through scripting.


MT: Do you have a favorite SolarWinds product?

 

MM: I would have to say the Engineer's Toolset is definitely my personal favorite although there are so many very neat and helpful tools offered by Solarwinds . There are certainly other tools and solutions but Solarwinds offers a very clean GUI that can quickly get you the answers you need when troubleshooting a problem and/or issue.

 

MT: What is your day job and how do you balance that with your blog activities?


MM: I work in Information Technology as a technical consultant for a large healthcare company. My day job gets my full attention from 8 to 5 so all my blogging activities including research, testing and actually writing any articles has to be done afterhours or weekends. Based on the exposure from my blog, I occasionally receive consulting solicitations which places even great demands on my free time.  I view blogging as a rewarding hobby which connects me with thousands of IT professionals around the world.

 

MT: You have any interesting hobbies you’d like to share?


MM: I enjoy playing ice hockey in the winter and watching the grass grow in the summer. I coach my twin girls' soccer team in the spring and fall seasons – a rewarding job watching all the girls as they have matured through the years in both in their skill and enjoyment of the game.

 

 

Other IT blogger profiles:

Ryan Adzima, The Techvangelist

Bill Brenner, Salted Hash

Tom Hollingsworth, The Networking Nerd

Scott Lowe, blog.scottlowe.org

Ivan Pepelnjak, ipSpace

Matt Simmons, Standalone SysAdmin

David Marshall VMBlog

I recently reached out to virtualization expert David Marshall in the latest of our series of blogger profiles.  In addition to being a really nice guy, David has developed a substantial following through his blog (VMBlog at www.vmblog.com) and content on InfoWorld as well as his books on virtualization.  So let’s get right to the discussion with David.

 

MT: You probably get this a lot, but how did you get started in IT?

david_marshall_cp.jpg  DM: I'm very much into gadgets and technology, and I've been banging away on computers since around 1978. My first computer was a Radio Shack TRS-80 Model III... and I was hooked from that point on. Oddly enough, I graduated from college with an accounting and finance degree, and did number crunching and investments for many years before I finally woke up and realized that you could actually make money and do a job that you were passionate about. So I left the financial world to take part in a new startup, an ASP (application service provider -- what we called cloud computing back in 1999). While trying to make the ASP business more profitable, I began experimenting with a new product in 2000 that was in Alpha code called VMware ESX and another Alpha product called Connectix Virtual Server (later acquired by Microsoft). And that was it for me! I was bit by the virtualization bug and never looked back. I helped create the first cross platform (or multi-hypervisor) virtualization management product around 2001, and continued to come up with new and interesting ways to use virtualization technology by starting a few more startup ventures to make those products a reality. It's been a blast! And I'm still working in virtualization 13 years later.

MT: And what made you take the leap to blogging about virtualization?

DM: This is such a fun question. I've talked about this with many people over the years, but I don't know how much I've ever really written about it. The short answer, if there is ever one with me, is that I started out blogging about virtualization because there weren't many people doing it. I started VMblog back in 2004, but that was a different time for virtualization. The technology was really still trying to prove itself, so it didn't really have books or blogs dedicated to it yet, and the user following was NOTHING like it is today.

So why did I start? One of my early startups was focused on server virtualization technology, and I began creating an internal email newsletter of sorts to keep co-workers informed about the technologies, new ways of doing things, and any other updates to the platforms we were working on. Back then, we were focused on VMware ESX, VMware GSX and Microsoft Virtual Server (remember those two platforms?). One of my co-workers told me, "Why don't you start a blog instead, and share that information with other people outside of the company?" That made sense to me, and VMblog was created. During that same time, I also started writing my first published book on the subject, which was a nice tie in to the blog.


MT: Who are your typical readers? What are they looking for from your blog?

DM: My readers come from all walks of life, from around the world, with different titles and backgrounds. But obviously, since the blog site is a niche, dedicated site, they all come for one thing -- virtualization. They are looking for the latest information within cloud computing and virtualization, whether server virtualization, desktop virtualization, or application, storage or network virtualization... or some form of cloud computing that relates to the technology. It's about information, education, events, whitepapers, books, trends, etc. It's about giving access to information and getting people in front of what's important or interesting. While niche, it still cuts across quite a number of technologies, and not everything in there will be for everyone, but it's a good place to start.  And over the years, I've opened it up for others to use it as a platform as well, if they didn't have one for themselves.


MT: After years of writing posts, what kinds of posts tend to be the most popular?

DM: Everything varies if you go by page views alone, and I'm sure it has ups and downs based on timing of when something gets published. But I can say that people pick up on the Q&A articles, and I certainly enjoy them because it's a lot of fun to have an opportunity to speak to someone and ask them questions about their latest announcements, news, products, or whatever. And I've also had a lot of success with my prediction series that I do every year. It's also a lot of fun for me, but I think it gets a lot of reads and people enjoy it because you get to hear directly from various experts and executives from companies both large and small, sometimes stealth or newly launched companies who may not yet have a voice, rather than just hearing from the pundits and analysts of the world as to what the coming trends will be or where they see the market or some specific technology headed the following year. The most recent series has come to a conclusion, but you can check out the 2013 predictions here:
http://vmblog.com/archive/tags/Prediction+2013/default.aspx


MT: I don’t want to steal too much thunder from that series but can you give us a high level view of what trends you see having the most impact on the virtualization space?

DM: That's a great question, and one that obviously can become a blog post all on its own. But a couple of quick hit things that come to mind include things like:

1. Platform Choice. It's fairly obvious to most people in the industry that VMware is the 800 pound gorilla -- the market leader. And with good reason, they have had the most stable and feature rich hypervisor platform for many years, and had virtually no real competition going back to its beginnings in 2000 when server virtualization was considered IT black magic! But things have changed with the introduction of the latest platform products from folks like Citrix, Red Hat and Oracle. And Microsoft's Hyper-V 3.0 probably is the biggest game changer.  Pundits have been talking about the hypervisor becoming commodity, and that trend into 2013 will become even more apparent. Imagine what the server virtualization market makeup would look like today if folks were just now getting involved with the technology. Hypervisors are closer than ever in feature and functionality, and price is becoming a big factor. Organizations have a real choice now. They don't have to choose between going with VMware or having to settle for something far inferior.

2. Virtualization Management is where it's at. We've seen it play out in 2012, and the trend of focusing on management will continue in 2013. As the hypervisor becomes more and more of a commodity and organizations continue to create hybrid environments, management software and management best practices will become more important to maintain a successful environment. With increased workloads being added into virtual platforms, cross vendor management tools and best practices will become critical to that success.

3. We're going to see the latest buzzword, the software-defined network or SDN, become an extremely important factor in the private and public cloud. As it continues to get defined, hardened, improved upon and accepted, we'll continue to see an emergence of new capabilities and a host of new startups playing in this market. And it will keep the server virtualization market on the cutting edge of things.


MT: What is your favorite SolarWinds product and why?

 

This is a really funny question, and the folks in the industry who know me already understands why that is... my obvious answer would be, SolarWinds Virtualization Manager.  In large part, because this product is my baby as it made its way to the SolarWinds family of products by way of acquisition from my startup company, Hyper9. 

 

But since the acquisition, SolarWinds has continued to expand on and improve the product, while still keeping and maintaining the product’s essence, which is why it has been so successful.  This virtualization management product really turned systems management upside down by throwing away the old, boring management paradigm of a tree view interface and moving into the 21st century with a modern, intuitive and scalable management interface that was designed on top of a search engine platform, making it perfect for a transient environment within a virtual datacenter.  And because the interface is widget based, it can change, grow and adapt to whatever the individual needs or wants rather than being forced to use an interface designed by a developer who has never managed a virtual datacenter.  And just like the management component, the built-in reporting and alerting also make use of the search engine design, so it's quick and easy to extend the product to perform custom and shared queries to build new reports and new alerts, without having to wait for a product update from the vendor.  And don't forget about the troubleshooting, change tracking, sprawl detection and capacity planning aspects of the product!  Very powerful, easy to use, and highly scalable.  And it just keeps getting better as SolarWinds continues to build out the product's feature set and update the interface.  I'm no longer associated with the product, but I still highly recommend that people at least try it out.  You guys offer a fully functional 30 day trial, and I think people will really enjoy the different experience that it offers.



MT: Given your experiences, do you recommend other people get into blogging?

DM: Yes, absolutely. IF you have spare time and a passion for whatever it is you want to blog about. To me, those are key. If you aren't passionate about it, it probably won't last very long. Blogging can be very time consuming, but rewarding as well. There are plenty of virtualization focused blogs out there, but current bloggers are always welcoming new bloggers with open arms. It's a great community of people, and I've made a number of good friends because of it. Now, when you go to a tradeshow like VMworld, it's like a reunion of sorts, because we bloggers may chat, talk or Tweet one another over the courser of the year, but it takes an industry event like this to bring us together face-to-face since we are all spread out across this globe... but held together in one common bond by a love of this technology that we all blog about and cover in some shape or form.


 

Other IT blogger profiles:

Ryan Adzima, The Techvangelist

Bill Brenner, Salted Hash

Tom Hollingsworth, The Networking Nerd

Scott Lowe, blog.scottlowe.org

Ivan Pepelnjak, ipSpace

Matt Simmons, Standalone SysAdmin

VMware view has been a major topic at VMWorld 2012 and one of the key issue issues that even VMware is acknowledging is that the infrastructure costs are still a barrier for broader adoption of virtual desktop infrastructure (VDI).  In particular the unpredictability of the storage IOps and the differences between peak activities and steady state makes sizing and purchasing storage for VDI very difficult.  Essentially, sizing for peak storage loads resulting from boot storms or upgrade, antivirus or other change operations can make the storage bill too expensive to make VDI practical for many companies.  On the other hand, undersizing the storage capacity can quickly result in a poor user experience and rapid dissatisfaction, especially in the adoption phase.   Caught between these two problems VMware has put some additional focus on bringing down the infrastructure cost barrier with two features – View Storage Accelerator and View Composer Array Integration (VCAI).  These enhancements will reduce some of the key problems customers see with doing VMware performance monitoring to achieve holistic VMware monitoring for their VDI environment.

 

View storage accelerator is a capability that is focused on acceleration of the read process for VDI. Essentially they put in-memory cache of between 400 MB and 2 GB of RAM in a host based solution in front of the disk-based storage.  The cache will hold the primary storage bits that the virtual desktops are accessing from a read point of view and takes that load off of the disks in the infrastructure.  Depending on what operations or scenarios were looked at the capability showed SAN performance management improvements for peak read IOPs by up to 80% and average IOPs by 45%.  Different scenarios resulted in smaller reductions but were still substantial. That change can really make a substantial reduction in the required storage capacity and capital cost making it easier VMware performance monitoring.


The second capability they have added is called View Composer Array Integration (VCAI). Essentially this is a different strategy that offloads much of the workload from the virtual system to the storage array where the cloning and snapshot technology can be very efficient. View can then access those newly created images to rapidly deliver services to the end users.  VCAI is part of View Composer and integrates with NAS storage partners using the vStorage APIs for Array Integration (VAAI).  EMC and NetApp are the only partners that currently provide this capability but they expect more to participate in the future.


With these enhancements VMware is addressing probably the biggest infrastructure hurdle people encounter when they move to VDI but they recognize that other challenges remain. When you reduce the storage IOPs as your primary bottleneck in these operations, that often exposes the next bottleneck, often CPU.  Additionally, while read IOPs storms are probably the most common, this won’t help with operations that are write intensive.  While there are more things needed to reduce the infrastructure costs, these enhancements should help VDI infrastructure budget requests make it past the laugh test. VMware troubleshooting simplified!

The Monday general session at VMWorld 2012 covered a lot of ground but didn’t really provide any major new technology announcements.  It started off with some of the traditional background statistics showing VMware’s and virtualization’s growth from 2008 to 2012 including an increase in virtualized workloads from 25% to 60% and that the key question regarding Cloud has changed from “What?” is it to “How?” do you implement it.  With that introduction, Paul Moritz went on to lay out his vision for the future, first with the general and pretty widely used paradigm that IT must provide services “Wherever, Whenever, and in Context” in the future. In order to make this transformation VMWare sees a number of transformations at the various layers in IT:

  • Infrastructure Layer: a transformation from server based to the cloud
  • Application Layer: a transformation from existing applications to new, cloud friendly applications and big data
  • Access Layer: a transformation from the PC to mobile devices.

While none of that is really new news, they then got into the heart of what they are aiming for with a “Software-Defined Datacenter” where all the infrastructure is virtualized and delivered as a service and automation is done by software.  While the concept isn’t new, it did lay out where VMware will be putting their emphasis in the near future trying to expand out from just the compute components to virtualized network and storage capabilities as well.  As part of this effort they announced the vCloud Suite which appears to be a regrouping primarily of their existing capabilities including management (vCenter Ops and vFabric), virtualization and cloud with vSphere and vCloud Director and adding new capabilities for software-defined networking and security plus software-defined storage and availability all available through a set of APIs.

After announcing the vSphere 5.1 release along with some shots at Microsoft Hyper-V around performance and reliability they got to what was probably the only really new news of the session that VMware will drop their current vRAM based pricing scheme and move to a per CPU price with no resource limits.   This was based on a survey of 13,000 of their customers and clearly is an effort to correct what has been broadly acknowledged as a tactical mistake on their part. 

The remainder of the session went into more details about how they will implement the vCloud Suite type capabilities including a couple relatively standard command line and screen-shot demos.  Tuesday’s general session is “Delivering on the Promise of the Software Defined Datacenter”, we’ll see if any exciting news comes out of that.

In my first VMWorld blog I asked the question “Will it be all about the Cloud again this year?”  From the first day of sessions and my take on the General session topic this morning I think we have an initial answer – while cloud hasn’t gone away, “software-defined” seems to be the focus from VMware.  VMware is already all about software defined computing with all their virtual computing capabilities but it seems the Nicira acquisition has them thinking about “software defined” everything.  Nicira moves them in the direction of abstracting networking capabilities from the hardware to the software layer.  They are also looking talking about “software-defined storage” as well. Basically, VMware’s goal is to control all of the major infrastructure components of a data center. 

 

This also represents a shift back from the “cloud will rule the datacenter” message that VMware has given in the past.  In reality this makes sense.  Cloud is a great concept but the implementation of real on-demand clouds is difficult.  It is not just about being able to automatically deploy another VM.  This shift could be seen as a tacit acknowledgement that there is still a lot of work to do to be able to effectively deploy all the services and capabilities required to run a data center with an on-demand cloud approach.  This is especially true when there are still hardware components that require configuration and management.  So the focus on software-defined infrastructure capabilities makes sense in a journey toward the more advanced automation that real cloud implementations require.  Cloud automation clearly exists today but it either requires some very heavy lifting to do full automation of the provisioning and configuration activities or you have to compromise on the scope and flexibility of your cloud (e.g., use predefined vLAN IP addresses, preconfigure and assign storage, etc.).  By first working on extracting the operational tasks for each of the infrastructure components to a software layer, you take the first step to really facilitating the automated deployment of those resources. While being a more incremental step towards full deployment and monitoring of the cloud, it also is an acknowledgement that the path may be longer than originally advertised.


The second major point to all this is that VMware’s clear goal is to own all of the underlying infrastructure components of a datacenter.  That is both thrilling and terrifying at the same time. On the thrilling side, VMware has enough execution muscle power that people in the industry have to take them seriously.  That is likely to drive many of the current players who make their living in the spaces VMware is targeting (e.g., networking, firewalls, storage, etc.) to accelerate activities that provide an alternative to VMware controlled abstraction of those infrastructure components.  At the same time, the idea of one vendor that abstracts and controls compute, networking and storage is a scary concept.  If you didn’t like the level of control Microsoft had with its operating system, just imagine what that would be like.

Filter Blog

By date: By tag: