40 Replies Latest reply on Dec 19, 2013 2:37 PM by deverts

    Would You Automate QoS Policy Based On Real-Time Flow Data?

    Ethan Banks

      Following up my query about how folks are making the most of their flow data, I’m also curious about what automated tasks people are performing as reactions to flow data, if any. This question is rooted in the reading & writing I’ve been doing about software defined networking.

       

      For example, let’s say there’s a 10Mbps WAN link we use for voice as well as data that a monitoring station is getting flow data from. In a statically defined network (i.e. the way we do things today), we’ve probably built a QoS policy for that link based on expected peak call volume, but the key is that the interface definition is static. We make our best guess as to how to provision the interface based on traffic mix history and expected future trends.

       

      But, what if the QoS policy was flexible based on current demand the flow data is telling us about? The idea is that the flow data gets exported to a controller that can reprovision the QoS policy of the WAN interface on the fly. More calls coming through? Beef up the low-latency queue. Fewer calls? Re-allocate queues to other flows. I recognize that there are products that can do this, although I think of them as service provider tools than enterprise tools for the most part.

       

      The question then is this: does running a network this way - where real-time flow data could change the overall traffic policy of a network – sound like the best thing ever, or more like a miserable pain in the backside you’d get sick of troubleshooting? I know networking vendors want to take our networks to this software defined panacea in a thousand different ways, so it’s an interesting point to discuss.

        • Re: Would You Automate QoS Policy Based On Real-Time Flow Data?
          wluther

          Sounds like a pain to me.

          As I have said before, I would rather make changes manually, although a semi-auto macro standing by would not seem to be a bad idea.  I would not mind a system to advise as to the best option at the time, perhaps showing the affects of the change(s) once made.  But when it comes to actually making that move/change, I want a non-Skynet employee in control.

          • Re: Would You Automate QoS Policy Based On Real-Time Flow Data?
            cahunt

            This could be good, but I would still want a base configuration in place to be able to switch the flip and go back to in case of issues.

            Having presets programmed in and adjusted specifically based on current traffic analysis would be better than giving the reins to your robotic arm.

            Let's hope the programmer programmed left as left.

            • Re: Would You Automate QoS Policy Based On Real-Time Flow Data?
              supermon

              I'm with , sounds like a nightmare to support. Advice is a great idea, but don't automate changing the stuff I have to support.

              • Re: Would You Automate QoS Policy Based On Real-Time Flow Data?
                rharland2012

                It could be useful - I think the ability to turn it 'up and down' in degrees would be essential, however. I like cahunt's idea of presets. If one could build the presets intelligently with knowledge of environment in mind, then you could automate that piece without fear of cybernetic overlord syndrome.

                • Re: Would You Automate QoS Policy Based On Real-Time Flow Data?
                  storn

                  In my opinion, this is why we use monitoring and alerting tools. Alert based on what ever the thresholds are and make the changes manually.

                  Dynamic management can be dangerous and unpredictable. Just ask the geniuses that implemented the auto-traders for the stock market.

                  • Re: Would You Automate QoS Policy Based On Real-Time Flow Data?
                    donpepe

                    Simply no. Your network confiugration should be very intentional. Having it change automatically is asking for trouble.

                    • Re: Would You Automate QoS Policy Based On Real-Time Flow Data?
                      mbwalker

                      I can see both sides of this. With today's dynamically changing network traffic I can see doing some automation to adjust Policy based on flow data but I also agree this could be very dangerous, unpredictable and, more importantly, difficult to troubleshoot. Forecasting traffic patterns would be even better. Alerting on flow data thresholds then modeling and forecasting this data would be more help than the automation. Even use of ToD/wk/month in your Policy (if available in equipment used) based on these models would better server the support community. I think donpepe said it best,

                      Simply no. Your network confiugration should be very intentional. Having it change automatically is asking for trouble.

                      • Re: Would You Automate QoS Policy Based On Real-Time Flow Data?
                        pyro13g

                        QoS policies can be flexible in that way.  A voice policy can be a priority queue for peak call volume but that doesn't mean other classes can't use bandwidth from the voice policy if available.   The QoS implementation on the router, for example, is already looking at the traffic and managing it dynamically based on your rules.  Making dynamic changes to network device configurations would never fly with our change management policies.  If your QoS config is changing and problems come up how would you ever know what the configuration was at the time of the problem?  Depending how frequently changes are made you could bring real-time change detection applications to their knees and possibly the network device too.  In environments like ours where QoS markings are set as close as possible to the source of the traffic, you're talking about potentially changing thousands of interface configurations on the fly.  No thanks.

                         

                        I'd rather let the QoS mechanisms on the device do their thing and have a traffic monitoring tool that presents me information on the traffic and my QoS policy performance.  Information like really good base lining for each metric/KPI, alerts when thresholds are reached and the duration, granular detail that is retained for as long as I want, baselines per polling/collection interval(not the entire 8 hours of my chart).

                        • Re: Would You Automate QoS Policy Based On Real-Time Flow Data?
                          vlan101net

                          I have a similar opinion to many others here.  I don't want QoS policy updated automatically based on real-time flow data.  I want alerts, possibly with recommended actions, but an engineer should review the data before acting.  I believe proper knowledge and design ahead of time for your QoS should cover the majority of your traffic patterns.

                           

                          Where I see flow data and SDN merging is in relation to historical traffic patterns.  I would be more comfortable with QoS templates being created based on history of traffic.  So during the standard month end processing in an enterprise a different QoS policy could be automatically be applied, then reverted back once that standard period had passed. 

                           

                          Service provider networks could be a different story.  With the volume of traffic and the need for great agility, full automation could be a useful tool, as long as the appropriate controls could be implemented to ensure all SLAs are still met.

                          • Re: Would You Automate QoS Policy Based On Real-Time Flow Data?
                            marc.coxall

                            I think this is how things will go but not until the technology is proven and there is a unified set of tools for doing it. Much like people in SME's were originally reluctant to move to dynamic routing protocols there is a feeling of a loss of control.

                             

                            It will take a culture change to have people trust a system more then they trust there own actions. And a culture change will take longer then the technology will take to change.

                             

                            For me I would dip my toe into it and implement it on a few connections i could get away and see how it feels.

                            • Re: Would You Automate QoS Policy Based On Real-Time Flow Data?
                              hulattp

                              Kinda echoing a few other peoples views, some automation would be handy, but not full blown control.

                              Maybe a few different plans, plan 1 - monday morning when everyone has forgot passwords, plan 2,3,4, 5 - friday afternoon when things are quiet as everyone is leaving to get home...grantallenby

                              • Re: Would You Automate QoS Policy Based On Real-Time Flow Data?
                                jswan

                                I'm kind of surprised by the generally negative response. I probably wouldn't implement the specific example you gave (changing the priority queue size), because I can't think of a situation where I would have a small priority queue, a number of calls that could burst above that size, and where I wouldn't use CAC.

                                 

                                As a general principle, though, I think the idea of using flow data to inform network policy dynamically is a great tool to have. The key point in your post is the concept of a controller -- we could write expect scripts right now that could read the flow cache and respond by changing policy-maps or installing policy routes via CLI stuffing, but that seems pretty brittle. If there's a controller more along so-called "SDN" lines that talks to the router via a sophisticated, programmable API, on the other hand, that seems like a great idea to me, albeit one that requires a slightly different view of network management. My big fear is the abstraction of the API into horrible GUIs, but that's a different topic...

                                • Re: Would You Automate QoS Policy Based On Real-Time Flow Data?
                                  deverts

                                  All,

                                   

                                  I've read many of the latest Ambassador postings about SDN, NSX, and automating the world. Do I think it's coming? Do any of us really have a choice? In another posting I made a comment about the 1990s .Com bust, how soon we forget the trouble that mess caused as the Big IT companies tried to market their services and then collapsed under the weight. It seems to me they are doing it again. And then it was all about outsourcing, and now companies are beginning to in-source again. History is doomed to repeat itself, because no one is learning from their mistakes. Technology changes so rapidly that no one gets to learn a lesson before making another.

                                   

                                  Cisco already has an Auto-QoS feature, and every engineer I've ever spoken to says stay away because it doesn't work right, the algorithms don't work correctly for every environment. I like the idea of an alert with "recommendations," but if I know my networks well enough, I already know what that recommendation should be. When you say "Real-Time Flow Data" are you referring to Netflow data? The default export on that is every 5 minutes, so by the time your monitoring tool detects an issue and adjusts, the issue might already be resolved, if it was a momentary data spike. In this case, basing QoS on "real-time" flow data is a bit misleading, and will create issues by reconfiguring your policies unnecessarily.

                                   

                                  I could go on and on, but this quote from Jurassic Park sums it up best:

                                   

                                  Dr. Ian Malcolm: If I may... Um, I'll tell you the problem with the scientific power that you're using here, it didn't require any discipline to attain it. You read what others had done and you took the next step. You didn't earn the knowledge for yourselves, so you don't take any responsibility for it. You stood on the shoulders of geniuses to accomplish something as fast as you could, and before you even knew what you had, you patented it, and packaged it, and slapped it on a plastic lunchbox, and now

                                  [bangs on the table]

                                  Dr. Ian Malcolm: you're selling it, you wanna sell it. Well...

                                  John Hammond: I don't think you're giving us our due credit. Our scientists have done things which nobody's ever done before...

                                  Dr. Ian Malcolm: Yeah, yeah, but your scientists were so preoccupied with whether or not they could that they didn't stop to think if they should.

                                   

                                  D

                                  • Re: Would You Automate QoS Policy Based On Real-Time Flow Data?
                                    gljb77

                                    Why is automation such a bad thing? We all see the benefits of it every day... I am all for it! As long as it is devised and configured correctly and easy to override why the heck not? 

                                     

                                    Whilst we are busy interpretting analysing and making changes we've missed our moment and that critical call that was taking place encountered loads of jitter and delay. The users calls complains, an incident is logged and now I've got more work to do.. Just what I need! my flow data identifies an issue, and it knows and I know what needs to be done, just make it so! 

                                      • Re: Would You Automate QoS Policy Based On Real-Time Flow Data?
                                        pyro13g

                                        You're not going to get the flow data quick enough to address the phone call scenario.  If VoIP is suffering at all your priority queue doesn't have a high enough bandwidth reservation or you need to implement CAC.  And be careful what you wish for. You may automate yourself out of a job!

                                        • Re: Would You Automate QoS Policy Based On Real-Time Flow Data?
                                          deverts

                                          Please don't misunderstand my comments, I like automation as much as anyone. And I'm a firm believer that our jobs will evolve over the next few years...for the good or the bad. But, history has proven that each IT environment is slightly different because each business does things differently. And while we are supposed to follow best practice designs and implements, there are very different opinions on what a best practice actually looks like. So, with all the possible variables out there, how can anything that must be "programmatically" scripted or coded ever automate what must be first analyzed? For that, you first need a tool that is possible of determining the true root cause of the issue. I've used almost all the monitoring tools out there, NetQoS, HP, CA, Nagios, etc...the tools are only as good as the person configuring them, and I've never seen one capable of "true" root cause analysis.

                                           

                                          So, I ask...how can we be expected to trust a tool to automate the process, when we aren't 100% positive it has chosen the proper corrective action?

                                           

                                          Also, here's a scenario for you. Let's say I want to configure my environment to increase bandwidth for VoIP when the line is congested, but another admin (could even be in the same shop) prefers to decrease bandwidth on the queue causing the congestion. Who's method is correct? Should the tool developer choose the right method? Should the tool do both and let us decide? The answer is...Yes, all the above, because everyone does things differently.

                                           

                                          The purpose of this blog is to provoke open dialogue and discussion, I'm not here to persuade anyone. Just offer my opinion, weigh in with my experiences, and hope that some day I can retire without being outsourced "again."

                                           

                                          Incidentally, has anyone looked at the Cisco training site for SDN? Has anyone noticed you must be a CCIE BEFORE you can take SDN classes? That doesn't sound like "easy automation" to me.

                                           

                                          D

                                        • Re: Would You Automate QoS Policy Based On Real-Time Flow Data?
                                          802jr

                                          I think going into this dynamic type of networking is good as long as the administrators are capable of decipherring what is happening dynamically. Yes, this should make our day-to-day network performance better, but I think as long as the engineer(s) behind it all are knowledgeable enough to turn it off and do what SDN would do dynamically.  In short, it is good for the network engineer to able to decipher everything that is happening on the network without trying to learn everything that is happening as dynamic changes occur.

                                          • Re: Would You Automate QoS Policy Based On Real-Time Flow Data?
                                            byrona

                                            I think this is one of those things that sounds good on paper but would turn out to be a pain in reality for those supporting it.

                                            • Re: Would You Automate QoS Policy Based On Real-Time Flow Data?
                                              wbrown

                                              As I've mentioned in other posts I'm one of those paranoid people that doesn't like systems making automatic changes to my configs.

                                              However, the scenario postulated is valid and I've seen a similar example used for automatic provisioning of virtual servers.  As I think about this I'm questioning my paranoia as I try to determine how could this process be any different than an IDS appliance blocking or rate-limiting traffic.

                                               

                                              My paranoia is winning out.  I'd rather have a scheduled script run to modify QoS policies for the expected traffic load during a given time period.  This allows for some tuning of the policies while still leaving the configs and operation in a known, or at least expected, state.  Trying to diagnose connectivity/latency/etc could be not so easy when the state of the traffic path is in flux.

                                              If a reaction is needed to deal with a lack of bandwidth I'd prefer to receive an alert and then determine next steps.  There may be a lack of bandwidth for a legitimate application or there may be other issues that need addressing.

                                              • Re: Would You Automate QoS Policy Based On Real-Time Flow Data?
                                                michael stump

                                                I was thinking about this over the weekend. I was listening to a freight train blow its horn as it headed toward a mountain tunnel. Two long, one short, one long. It's a well-understand warning that a train is coming through.

                                                 

                                                What if large data transfers could do the same? If applications could announce their intent to move lots of data, could the network autoconfigure itself to expedite the transfer, then return to its previous state once the transfer was completed? I suspect the arguments against this are in line with traditional, "I don't want to automate important stuff" gripes that eventually fall away (remember the grumbling about fully automated DRS a few years ago?).

                                                • Re: Would You Automate QoS Policy Based On Real-Time Flow Data?
                                                  petersjd

                                                  Only after a firm baseline has been established and not without thorough testing.  There will always be human factors that can cause an exception to rules and thresholds that the system would not be aware of.