21 Replies Latest reply on Nov 5, 2008 10:09 AM by unclegio

    VMware – managing virtual switches inside your ESX Server

    denny.lecompte
      From talking to customers, we know that many-possibly most-of you work in organizations that have embraced virtualization to some extent.  VMware's ESX server is especially popular.  One interesting aspect of ESX servers that hasn't gotten a lot of attention are the virtual switches inside that ESX server.  We here at SolarWinds are wondering what the Thwack community thinks about virtual switches and virtual switch management.
      • As network engineers, how do you manage those virtual switches?
      • As network engineers, do you monitor virtual switches?
      • What visibility do you have into the network traffic inside the ESX server?
        • Do you wish you had more?
        • Have you ever experienced problems related to traffic inside the ESX servers?

      Virtualization is only going to grow, and we'd like to hear your thoughts on how its impacting networking and network engineers.

        • Re: VMware – managing virtual switches inside your ESX Server
          ceclark

          Denny-

          It's a good thought, but from what I understand, the VM switches are generally dumb. They aren't configurable (like a physical switch), aren't manageable, they don't pass STP, etc. I have never experienced an issue with the virtual switches, but our VM guy is pretty good so he knows how to configure them. Even on my test system, I've never had a problem, but I have not purposely tried to break it either. I'd be interested in hearing if/how we can monitor them.
           
          Collin 

          • Re: VMware – managing virtual switches inside your ESX Server
            JeanC.
            We currently are using VMware's ESX Servers.  We are not monitoring the virtual switches.  Should we?
            • Re: VMware – managing virtual switches inside your ESX Server

               I personally would like to have a better handle on the traffic that flows across the vSwitches in our VMware environment. We have a number of servers that our application staff swears is our fault for performance. We are looking at additional tools to dive into the server and disk systems in VMware, better network information would be beneficial as well.

               As an aside, if the APM also provided information about the VMware environment, that would be beneficial. Again, our application developers have figured out how to write code in such a way that 100% of the fastest machines on the planet are still not enough to handle their code.

               Details include, vCPU time assigned, Real CPU time, I/O wait time, Memory stats such shared pages, the ballon. On top of all this the reservation vs available statistics.

              I would really need to sit down and think about what stats would be the most important. I bet if we had them all I could find a nugget of useful information like I did with Orion and Netflow when I discovered the 2 to 3GB file copies across a T-1 every time a person logged on.

              • Re: VMware – managing virtual switches inside your ESX Server

                We have several VM ware server here with multiple VM switches.  We would like to see up/down, traffic flow, packet loss to them.


                 


                Yea I can see the guest servers and their nic's but seeing this info for the host would be great.

                • Re: VMware – managing virtual switches inside your ESX Server
                  epenney

                  Physical servers with dedicated networks ports are rapidly becoming a thing of the past. On a daily basis our organization is virtualizing its physical servers. This is making network services job of providing feedback/performance monitoring very difficult. Any and all visibility, performance reporting that Orion can do for ESX infrastructure is so desperately needed.

                  • Re: VMware – managing virtual switches inside your ESX Server
                    SamuelB

                    I would like to see at least the same information for virtual switch interfaces that we can currently see with physical switch interfaces in Orion. Being able to identify the VMs that are the top bandwidth users would assist me in telling the Server Admins that the network is not the problem. We haven't experienced any problems yet, but I just know (based on history) that it is only a matter of time.

                    • Re: VMware – managing virtual switches inside your ESX Server
                      Indeed, I agree, I think it would be phenomenal to be able to get statistics from the VM Switches.

                      Then I'm guessing that VMWare would have to create some form of management access to them.

                      To answer your questions:

                      1) The server teams manage the VM Switches.
                      2) Network teams don't monitor them.
                      3) No visibility.
                      4) Yes wish Network Teams were able to look deeper into them.
                      5) Yes.. Had to monitor at the aggregate port on the Cisco switch that the server was connected to.

                      I personally think we need to see the VM Switches managed like any other switch on the network, with relevant security controls, port mirroring etc. that could be enabled.
                        • Re: VMware – managing virtual switches inside your ESX Server
                          craig999

                          Thats a great idea. We would love to see it.

                            • Re: VMware – managing virtual switches inside your ESX Server

                               Just an FYI, you can enable netflow on the virtual switches...I haven't done it yet

                               

                               

                              http://www.vmware.com/pdf/vi3_35_25_netflow.pdf

                               

                              Thanks,

                              mark

                                • Re: VMware – managing virtual switches inside your ESX Server
                                  denny.lecompte

                                   Yes, you can.  It's still listed as "experimental", but we are working with VMware on their efforts to make the support official.

                                    • Re: VMware – managing virtual switches inside your ESX Server


                                       Yes, you can.  It's still listed as "experimental", but we are working with VMware on their efforts to make the support official.

                                       

                                       

                                       

                                      Yeah, I found out it is definitely experimental. Last month I tried it on one of our VMware hosts and well, it didn't go well. Thankfully I moved off all our production stuff and just left up a test workstation on it. Ultimately I had to one finger reboot it because the system refused to stop the service.

                                        • Re: VMware – managing virtual switches inside your ESX Server
                                          tigershark

                                          We are at the very early stages of virtualization. We were really early adopters in my company - have been using virtual servers since 2003 - but It is definitely where is going. A few years from now standalone servers will be practically gone. It would be nice to be able to monitor every aspect of VMware since, as we all know, it behaves very differently. 

                                            • Re: VMware – managing virtual switches inside your ESX Server

                                              We have a virtualization project underway and this type of monitoring would be very beneficial but I am not as concerned about the traffic between machines as our machines VMotion all over the place this data would be irrelavent but the vCPU and vMemory and other Host/Guest machine specific information would be much more beneficial.


                                              Right now VMWare's Virtual Infrastructure client is better than Orion since Orion doesn't exactly know how VMWare is doing the resource sharing but I am glad to hear you are looking into monitoring Virtual environments..

                                                • Re: VMware – managing virtual switches inside your ESX Server
                                                  William Vitalec

                                                  Has anyone ran in to issues with everything being in a virtual environment?  I have one environment that has 3 pollers, a HSB and SQL Server 2005.  They are all on different hosts.  I have issues every day with the pollers falling behind and devices and interfaces left not being polled within each polling cycle.  Some of them can fall hours  behind.  Is my SQL Server being on a physical box better?  If so can you tell me specifically why and what is affected by it being on a virtual box?  Also, if someone has had these issues before with the same type of set up, what did you do to correct it?  How did you know it was fixed?  Do you monitor this with another application or did you just query the DB to find out how many devices/interfaces were behind?

                                                    • Re: VMware – managing virtual switches inside your ESX Server
                                                      denny.lecompte

                                                      Has anyone ran in to issues with everything being in a virtual environment?  I have one environment that has 3 pollers, a HSB and SQL Server 2005.  They are all on different hosts.  I have issues every day with the pollers falling behind and devices and interfaces left not being polled within each polling cycle.  Some of them can fall hours  behind.  Is my SQL Server being on a physical box better?  If so can you tell me specifically why and what is affected by it being on a virtual box?  Also, if someone has had these issues before with the same type of set up, what did you do to correct it?  How did you know it was fixed?  Do you monitor this with another application or did you just query the DB to find out how many devices/interfaces were behind?

                                                      I can tell you that our observation of many customers is that having that database on a virtual machine tends to hurt performance. The primary reason seems to be  that read/write is slower.  If you had SQL virtualized but writing to a physical hard disk, it would probably help.  That's just our observation.
                                          • Re: VMware – managing virtual switches inside your ESX Server
                                            unclegio

                                            Cisco has a product comming out that will replace the current VMWare VSwitch.  Then you can actuall hand off your networking at the virtual switch to your infrastructure team and just focus ont he servers (hosts and guests).  the Cisco line will be called Nexus.

                                        • Re: VMware – managing virtual switches inside your ESX Server
                                          jdedmundson
                                          We are looking at Reflex Security http://www.reflexsecurity.com/ but we would like to monitor the virtual switches within Orion. As a network engineer, we are having a hard time managing the virtual switches. Server team is not too happy right now because we will not trunk all the VLANs to them. Because we just not for sure of how it will route.
                                          We would like to monitor the virtual switches within Orion.
                                          We do not have any visibility within the virtual network unless we use the Reflex Security.
                                          Yes we want more.

                                          Our biggest concern is that the server team will configure one of the virtual switches wrong and we could end up with routing problems. As it stands now they have to ask the Network team for the change, we make sure that the change will not conflict with any thing and after the change we monitor the traffic from the change to make sure we do not have any problems. But with virtual switches that is out of the Network team hands.

                                          Our Tech Support Team likes Orion very much and they like that they can go to one system now and see everything that is going on with both the servers and the network.

                                          So please Solarwinds give us the tools to look into the virtual world.
                                            • Re: VMware – managing virtual switches inside your ESX Server
                                              denny.lecompte

                                              We have spoken to VMware about this very issue several times. Unless they expose the APIs for us to manage their switch, there is little we can do.

                                               They announced their strategic approach at VMworld 2008.  They intend to allow vendors to create their own virtual switches.  And Cisco has first to announce a virtual switch for VMware ESX Servers.  It's called the Nexus 1000v, and it runs their NX-OS and should be manageable just like any device running the nexus OS.  It supports SNMP, NetFlow, etc. 

                                              I don't want to push Cisco's product. You can learn more here.  Other vendors will certainly come out with similar solutions for  VMware.  This is VMware's direction and what they intend to support.  We'll work with them on ths strategy.