41 Replies Latest reply: Jan 2, 2014 1:24 PM by ttrudeau RSS

Network Sonar Discovery - Purpose of Interface Discovery

karltbraun

Can someone enlighten me as to the purpose of changed interfaces showing up in the discovery screen?  There doesn't seem to be anyway to take care of changed / newly discovered interfaces other than to ignore them.  If I select them and import, they get ignored because the node is already in the database.  If I have to ignore them every time, why not just automatically add them to the ignore list?  And if we are just going to ignore them, why "discover" them in the first place.  This is creating about 45 minutes of work a week just selecting and ignoring changed and discovered interfaces.

 

Thanks for any help on this.

 
    • Re: Network Sonar Discovery - Purpose of Interface Discovery
      Tomas Mrkvicka

      Hi Karl,

      could you please open support ticket for your issue and paste number of ticket to this forum? For us it will be easiest way how to help you.

       

      Thanks.

      • Re: Network Sonar Discovery - Purpose of Interface Discovery
        rgward

        Tomas, there are others that would like to know the answer to Karl's question as well.  Please post the response so we all know the answer to the mystery.  Thanks.

      • Re: Network Sonar Discovery - Purpose of Interface Discovery
        smartd
        Yes.  This is a long standing complaint with Discovery.  I want "found" nodes, not "changed" interfaces. 
        Found should be default.  And you should be able to "Ignore" by device type, using the static and dynamic  filtering features available in the new "groups" feature.
        • Re: Network Sonar Discovery - Purpose of Interface Discovery
          Tomas Mrkvicka

          We still don't know about any support ticket for this issue. If there is anybody who can reproduce this issue please feel free to open support ticket. After that we can start investigation directly at customer's side and then inform others in this forum about results.

           

          Please don't forget enter ticket number into this forum in order we and other users know it is under investigation already.

           

          Thanks

      • Re: Network Sonar Discovery - Purpose of Interface Discovery
        karltbraun

        I will be opening a ticket on this this afternoon (WED-11-JAN) or tomorrow.

        • Re: Network Sonar Discovery - Purpose of Interface Discovery
          DanielleH

          Hi karltbraun--

          Did you ever open a support ticket?  If so, would you please post your ticket # and any progress you've made in the process?

          Thanks!
          DH

          • Re: Network Sonar Discovery - Purpose of Interface Discovery
            karltbraun

            I have opened Case # 304991 for this.

             

            There seem to be two aspects to this; the first doesn't really warrant a ticket:  What was the thinking behind the way this designed to work?  What was the intended maintenance process?

             

            The second part is probably a feature request: is there a way to, on a per-node basis (possibly mass-configured by certain attribute criteria) the behavior of discovery of changed interfaces?

             

            The problem is that for many nodes, when the system is rebooted, the interfaces change, and if you don't wish to monitor the interfaces, you have to check each one to ignore it.   Also: If you have (for example) a switch with a large number of ports, but most of them not critical (so we don't monitor the interface), you have to first discover the switch and import the node, list the resources, turn off the management of the interfaces, then on the next discovery go in an ignore all of those interfaces again.

            Surely SW engineers face this in daily operations.

              • Re: Network Sonar Discovery - Purpose of Interface Discovery
                karltbraun

                I've generated the requested diagnostics file.  It is 58.4MB.  Please instruct on how to get this to support staff.

                -ktb

                • Re: Network Sonar Discovery - Purpose of Interface Discovery
                  Tomas Mrkvicka

                  Instruction for sending diagnostics are included in your support case. If there were some issues with please let me know here in forum.

                  • Re: Network Sonar Discovery - Purpose of Interface Discovery
                    mgibson

                    In 10.2.1 I too are having the same issues with importing/ignoring the discovery results.

                    If I choose to import a changed or new interface, I get an error stating the import skipped because it exists in the DB. The Node exists for sure, but the interfaces in question are not being monitored, therefore this points to a possible coding issue in the application.

                    As for "found" nodes, I see nodes showing in the discovery that are being monitored allready and they are being marked as "found". I think it has something to do with it not looking at all nodes on all pollers.

                    This is definitly broken, I feel this is a major issue for me, I have case #306812 that was opened with a medium priority on yesterday and still have not recieved a response yet.

                  • Re: Network Sonar Discovery - Purpose of Interface Discovery
                    karltbraun

                    First, due to a problem reported over a year ago that no one has done anything about, the portal shows no cases for me.  Nothing open, nothing closed, nothing in progress - nothing.  Maybe there is a way to track this down, but I don't have time to debug your portal for you.

                     

                    Second, the instructions I had said to contact my support person for instructions on how to upload the zip file if it is over 5MB.  This is the only way I know how to do this.  Your comment is less than helpful in this regard.

                     

                    Third, this issue has been opened since NOV 18.  All I've been asking for is an explanation of how one is supposed to handle this.  This issue takes up a ton of time at least once a month with interfaces being rediscovered and having to be re-ignored.  I have no word on whether this is a bug or a process problem.  But I do feel like I'm getting the run around.

                     

                    When people ask me (as a consultant specializing in NMS/EMS) what I think of Solar Winds, I tell them the product is pretty good for the price, but you better dedicate at least one staff member to maintenance, because you're not going to get much help through support.  I say that because that's been my experience.  I've spent way too much time on this issue.

                    • Re: Network Sonar Discovery - Purpose of Interface Discovery
                      mharvey

                      Karl,

                      Our technician working on case 304991 requested the diagnostics on behalf of our developers.  They would like to collect that information from you in order to get the explanation to you that you are asking for.  There should have been upload instructions for the diagnostics file sent to you in the last email the technician sent.  As soon as you can get these to us, we can get the files back to our developers to get the information you are looking for.

                  • Re: Network Sonar Discovery - Purpose of Interface Discovery
                    karltbraun

                    I received the following email from Support:

                     

                    Re: Network Sonar Discovery - Purpose of Interface Discovery
                    By mharvey in Orion Network Performance Monitor

                    Karl,

                    Our technician working on case 304991 requested the diagnostics on behalf of our developers.  They would like to collect that information from you in order to get the explanation to you that you are asking for.  There should have been upload instructions for the diagnostics file sent to you in the last email the technician sent.  As soon as you can get these to us, we can get the files back to our developers to get the information you are looking for.

                     

                    The only instructions I have received are as above "included in your case", which I have no access to or otherwise have information on.  Is it asking too much to have someone actually contact me, either in an email conversation that I can reply to, or by phone, to provide these instructions?  I really feel like I'm wasting my time here.

                    • Re: Network Sonar Discovery - Purpose of Interface Discovery
                      systemstate

                      I have to agree with you Karl.  Honestly, I'm getting frustrated just watching this process continue.

                      I don't understand why you have to go through all this for a simple explanation of why a feature is designed the way it is.

                      By all accounts, the software appears to be working as designed.  I personally don't understand the purpose of that design- which is why I've been following this very painful thread.

                      Seriously-  Can't support just answer the question?  There are plenty of "why does this feature work this way" questions on here all the time that don't require people to submit tickets and support bundles.

                      • Re: Network Sonar Discovery - Purpose of Interface Discovery
                        mavturner

                        Karl, I have emailed you directly offline to provide the steps to upload your diagnostics to me. I will make sure they get to development.

                        If we knew why this was happening we would certainly provide why it is. Unfortunately we don't have enough information at this time. Once we know what the problem is, we will definitely explain why it is happening.

                        Mav

                      • Re: Network Sonar Discovery - Purpose of Interface Discovery
                        smartd
                        Let me take a shot at this.

                        We do NOT want auto discovery to detect interface changes.  We want an option to DISABLE THIS.  We also want the default for discovery to show FOUND ITEMS.  We do NOT want to see changed items by default.  This is a design problem, not a bug.

                        We also need device filters, but I'll drop that on the wish list site, which I'm going to carpet bomb in 2012.

                        • Re: Network Sonar Discovery - Purpose of Interface Discovery
                          karltbraun

                          =- CoSigned -=

                          • Re: Network Sonar Discovery - Purpose of Interface Discovery
                            cleahy

                            I'm having this issue as well.  I'm just learning SW, and I'm taking over an installation that was put in place by someone no longer in this department.  The system has been neglected for awhile and now I'm stuck with over 900 found/changed/imported items listed in the discovered node section, and no way to wean those down to what I need to check to see if they're already in the database.  (Not without checking over 900 entries one by one, against a list of thousands.)  This is a major stopping point for me in getting this installation useful again.

                        • Re: Network Sonar Discovery - Purpose of Interface Discovery
                          Tomas Mrkvicka
                          Let me take a shot at this. We do NOT want auto discovery to detect interface changes. We want an option to DISABLE THIS. We also want the default for discovery to show FOUND ITEMS. We do NOT want to see changed items by default. This is a design problem, not a bug. We also need device filters, but I'll drop that on the wish list site, which I'm going to carpet bomb in 2012.

                           



                           

                          Hi,

                          so if I understand it correctly you want following:

                          • Option to disable detection of new sub-elements (interfaces,volumes) for already imported nodes. Please notice there is nothing like "changed" interfaces - interface is either existing or new. Therefore if you'll have already imported node, this node gets new interface and you have enabled this new option then nothing will be shown. Is it correct?
                          • By "default for discovery to show FOUND ITEMS" do you means by default choose "Found" status in Scheduled Discovery Results?
                          • What would you expect from device filtering? Can you please describe real expected scenario useful for you? It can help us understand how we could design this feature in next releases.
                          • Re: Network Sonar Discovery - Purpose of Interface Discovery
                            cleahy

                            I just want a decent way of dealing with new nodes/interfaces.  We're a FAIRLY large shop, and our department is responsible for the backbone of the network.  We don't want to monitor everyone's printer, or desktop ports, we just want to monitor our electronics, and their uplinking ports.

                            What I'm faced with when I look at the Discovered Nodes section is completely unusable.  I need to see anything new, so I can go through it and decide whether or not to import it.  What I SEE is a list of over 900 nodes/interfaces, some new (although the already exist as managed node), some changed (although I can see no changes, and they too already exist as managed nodes), and some "imported".

                            That's useless information to me.  Why would I want a list of imported items, what I'm looking for are the ones that have NOT been imported, so I can go through them.  And why is there no way to isolate the items that need my attention, and why when I import a switch and all it's interfaces does it say "not imported", then STILL show in my list as "new" no matter what I do?

                            And although the admin guide tells you to "click here", and "click there" to import a node, the actual explanation on what all of these items are, and how to USE this list to actually keep UP with what's happening on my network is completely missing.

                            So, yeah.  I could use some help on this one.

  • Re: Network Sonar Discovery - Purpose of Interface Discovery
    systemstate

    Can we get an answer on this from the SolarWinds staff?  If it's a feature that we don't fully understand and aren't taking advantage of, I'd like to know.  Otherwise I have to agree with kartbraun in that it's exceptionally annoying.

     

    Thanks,

         Systemstate

  • Re: Network Sonar Discovery - Purpose of Interface Discovery
    GoldTipu

     

    The discover is seeing these devices as new, as it's actually rediscovering the same subnet/IP Range/etc.  The reason these are marked as new is because to the discovery they are new.  The discovery isn't checking against the database for nodes during the discovery process.  It only checks against the database when the import is attempted.  At that point it will state the node already exists in the database and move on.  The only way is to keep these nodes from being detected or rediscovered as new each time at the moment is to add then into the ignore list.  This will prevent the discovery from finding them again.  Please note, the discovery isn't intuitive when it finds devices so it things all devices are when it runs as they are responding to it.

    http://www.solarwinds.com/documentation/Orion/docs/OrionAdministratorGuide.pdf and start reading on page 42.

    • Re: Network Sonar Discovery - Purpose of Interface Discovery
      karltbraun

      I'm sure that's mostly true, but after you try to import these newly discovered interfaces, and it checks the database and says "nope, already in there', it still keeps the newly discovered interfaces as waiting to be processed.  As far as I know, at this point, you have two options:  (1) keep them in the waiting-to-be-processed state in perpetuity, or (2) select them all and ignore them.

       

      So far SolarWinds has asked me to run a number of scripts and provide them with the result.  That's fair when trying to resolve a bug that's unique to my system.  And to be fair - I have had time in the past week or so to do this.  

       

      But my question was: *what was the design - how does SW expect this process to run*? 

       

      This has been pointed out a number of times on this discussion.  I've yet to hear anyone try to address this.  Here's my experience with this process:

       

      (1) New switches and servers are discovered during a nightly scan.  

      (2) The nodes are checked off in the newly -discovered page, and imported into the database.  Then I configure the attributes so they show up in the right managed nodes containers.

      (3) I also list the resources on these and leave checked only the interfaces I want to manage.  (a) For switches, these are only interfaces that have uplinks/downlinks to other switches, routers, or firewalls, or critical nodes;  (b) For servers, we use Microsoft SCOM for most server management and alerting, so I uncheck all interfaces.

      (4) During the next nights run, for all of the nodes which I imported the previous day, all interfaces which I unchecked (don't wish to monitor) are then *REDISCOVERED* as if they were new.  I can't processes these by just checking the node - I will just get a message indicating the node is already in the database during the import process and the interfaces will still remain waiting to be processed individually.  So I have to go through and check each "newly discovered" interface individually and then set them to be ignored.  

      Imagine doing this on a 144 port distribution switch with 1 uplink and 6 downstream switches.

      For servers it is even worse.  All physical and logical interfaces on the server are "rediscovered" and must be checked and ignored the following day.  To make maters worse: If the server is rebooted and the interface index changes on the interface - they are all rediscovered again and I have to repeat this process... every time the server is rebooted.

       

      This *can't* be the way this was designed.  Surely SW engineers are involved in this kind of maintenance process and have a way around this, or this issue would be resolved by now.  Can someone from SW please get Josh or some other engineer to discuss this?

      • Re: Network Sonar Discovery - Purpose of Interface Discovery
        GoldTipu

        Orion will re-discover interfaces basses on the interface index . If you have added Interfaces in ignore list and the interface Index changed on the NODE  it will rediscover as NEW interface . 

        You need to make your devices to keep "Interface index persistent" 

        • Re: Network Sonar Discovery - Purpose of Interface Discovery
          mavturner

          Karl,

          I appreciate you sticking with this. This was certainly not intentionally designed to be such a pain for our users. It's clear we could have done a better job of getting feedback when the re-discovery process was built. To that end, if you would like to jump on a Go To Meeting with myself and some of our developers, we would like to go through this with you step by step so we can work on a fix that makes your life easier (at least where this bit is concerned). Yes, we can re-produce this internally and make it work the way we think it should work, but the goal is to make it work for customers and your feedback is certainly appreciated. I will email you off of thwack to arrange this.

          Regarding a better filtering of interfaces and nodes on discovery. That is a well know request (filter by description matching, interface type, etc. ). We won't be able to get this fixed in the release we are actively working on, but it's a high priority to get added soon.

          Thank you,

          Mav

          • Re: Network Sonar Discovery - Purpose of Interface Discovery
            karltbraun

            Is there any time frame for a change to discovery which will address these problems? I've spent about an hour during each of the past two weeks just clicking individual interfaces and then ignoring them.  Just to reiterate:

            • A lot of the problems stem from changes to interfaces with non-persistent interface indexes
              • This seems, in my environment, to be relevant to mostly servers, where indexes change after a reboot (and for Windows boxes, that's at least weekly); it also applies to some Colubris (Procurve) wireless management modules (WSMs/MSMs) where the indexes appear to change dynamically with the connection/disconnection of clients.
            • If, after a node is discovered, we could indicate (say, through a check-box) that we don't care about interface changes (but we do care about other changes SNMP might discover).
              • I wouldn't mind doing this on a per-node basis at time of initial discovery / integration

            Something Solarwinds developers may not get with respect to the way EMS/NMS systems work in large network environments: 

            • Activities are broken into the following categories (may be different groups)
              • Project / Implementation
                • This would include intitial discovery over a new network segment, etc.
                • The costs associated with this would be considered part of a large capital project
              • Operations / Maintenance
                • This is just plain overhead.  Management is going to be interested in keeping these costs (including head count) very low; it's part of the selling point of a NMS/EMS that capital investment in something like Solarwinds will reduce ongoing overhead costs (doing more with less people)
                • Of course any system needs ongoing maintenance.  But having to devote 10-15 minutes every day to needless interface cleanup of servers reduces the perceived effectiveness of the investment in the product.
                • If this perceived needless overhead is associated with, say, servers, its going to reduce the ability to justify using this product for anything other than simple network up/down monitoring.  That puts you in a class with What's Up Gold.

            So if setting the "ignore" check box during initial discovery is available, it becomes an implementation activity, not a maintenance/overhead activity.

    • Re: Network Sonar Discovery - Purpose of Interface Discovery
      rgward

       Please note, the discovery isn't intuitive when it finds devices so it things all devices are when it runs as they are responding to it.

      The discovery isn't checking against the database for nodes during the discovery process.  It only checks against the database when the import is attempted.  

      This is what needs to change!  The discovery needs to be more intuitive and compare each discovered node against the database on nodename and ip address. I would like to see an option to compare using the node short name or FQDN.

      On the front-end add a step to be able to select discovery filter criteria (vendor types, machine types, interface types, volume types, etc.) plus a checkbox to drop duplicates.  On the back-end, after all nodes have been discovered, match against the database on nodename and ip address and drop the matches from the discovered nodes then filter the remaining nodes by the additional user discovery criteria that was provided on the front-end before presenting the final import result.  The final result should be able to be saved for later import. 

      I also would like to see the ability to run concurrent discoveries across multiple polling engines.

      That's the way I'd like to see discovery work.

       

      Oh, I almost forgot, I would like to see discovery resolve a node discovered with multiple ip addresses to the Loopback address (if one is discovered).  This would be another discovery filter option (checkbox) on the front-end.

  • Re: Network Sonar Discovery - Purpose of Interface Discovery
    HemiTruck

    you need to stop continuously ignoring these as they pop up.  All you are doing is filling the ignore table with more **** that will never purge.

    learn how many are normally there for the windows devices and just pass over them.

     

    The problem stems from windows servers coming up from reboot with new interfaces that dont get assigned to the similarly named one that went down.  having it give you new interfaces is a good thing for instances where you add a module to a switch that wasnt full then you can import all the new interfaces.  only devices that seem to be causing issues with this is the windows devices that change the uuid of the interface when it reboots each and every time.


    • Re: Network Sonar Discovery - Purpose of Interface Discovery
      karltbraun

      I hadn't thought about that, but that makes sense.  Of course the DB is going to be filling up with obsolete interfaces.

       

      But I'm afraid your solution is not practical.  If I understand you, you're telling me I need to memorize those node/interfaces that are repeating and just don't process them in the Scheduled Discover results.  If I did that, but the end of a week, I'd have a growing list of over 100 "discovered" changed interfaces, and this would grow exponentially:

      • 100's of Windows servers in the environment, with dozens rebooted weekly.
      • Interfaces on HP (Colubris) MSMs change daily - sometimes a dozen interfaces change in a week, and I have around 100 MSMs in the environment.

       

      SolarWinds staff: this is going to get critical if I start running out of database room. 

  • Re: Network Sonar Discovery - Purpose of Interface Discovery
    smartd

    For example:

    I have 5227 nodes in my ignore database

    I am monitoring 2498 nodes with 4487 interfaces.

  • Re: Network Sonar Discovery - Purpose of Interface Discovery
    neil.broadley

    Well, this issue was opened in Dec 11, and we're fast approaching Dec 13, I'm running NPM 10.5, and I still get this exact issue.

     

    Even on devices with "snmp ifmib ifindex persist", we still see occasional changes to interfaces, particularly relating to VMware interfaces. Very frustrating having to ignore these over and over.

     

    I wouldn't mind if I could say to Orion - "go ahead, import these changes you've found, I'm fine with that". But no. When I try that, I get "node already exists, skipped", and I'm dumped back at the discovery screen with all these magically changed interfaces waiting to be imported/ignored.

     

    I wouldn't even mind if I could "select all" on the interfaces to ignore them. But no. I have to tick each one individually, tediously.

  • Re: Network Sonar Discovery - Purpose of Interface Discovery
    ttrudeau

    Bumping this thread.  This is my pain every.single.day.