48 Replies Latest reply on Nov 23, 2012 2:57 AM by thamizh85

    Connect Now Topology Discovery Problems

    pyro13g

      Is having a lot of problems figuring out topology in our network.   Anyone else?  I really haven't found a common cause.  Re-discovery in Network Sonar doesn't help.

      It doesn't appear to be able to figure out device connections in instances where SVI's are used.

      Failed to discover connectivity over layer 3 port-channels

      Failed to detect connectivity for point to point layer 3 connections.

        • Re: Connect Now Topology Discovery Problems
          aLTeReGo

          Is connect now Layer 3 or Layer 2? I thought it was suppose to be Layer 2.

          • Re: Connect Now Topology Discovery Problems

            pyro13g,

            aLTeReGo is right. We are looking at layer 2 connections.

            On your layer 2 connections that are not showing up, try to check that you have the bridge mib turned on. Check that you can read OID  1.3.6.1.2.1.17.7.1.2.2 or 1.3.6.1.2.1.17.4.3 from your Orion server. If it is turned on, you can check if the MAC address of the device you are connecting to is listed there. If you cannot find the MAC there, we are not going to be able to draw connections for you in this version. We are looking to add support for more complex scenarios but it will take us some time.

            As for the issue with multiple VLANs, we are looking to add support for that asap. We will let you guys know when we make it available.

            Hope that helps,
            W

              • Re: Connect Now Topology Discovery Problems
                ecklerwr1

                @will

                It looks like I have it here if I'm not mistaken... I will verify on my other routers... I'm still not seeing anything under the NPM Topology resource...

                I do have a case open 156592.

                • Re: Connect Now Topology Discovery Problems
                  jodros

                  after readin this thread, i guess now i understand why my layer 3 connected devices are not working properly with connectnow.  however, what about layer 2 connected devices over an etherchannel?  most of my network is not mapping properly with connectnow. 

                  also, it might have been mentioned before, but if there is a network device with more than one management IP address accessible to Orion, why does it add this device more than once?  why isn't it able to determine it is the same device as LANSurveyor does?

                  thanks for the quick responses and work on this.  so far v10 is nice, especially the VM support.

                    • Re: Connect Now Topology Discovery Problems

                      @jodros My guess about the etherchannel is that it is not in the bridge mib. You can use the switch port mapper in toolset or some other mib browser to check. In general, we only see physical connections there and the etherchannel seems more logical than physical.

                      Currently we do not support multiple IPs for a single node. We know this is a limitation and we are looking for an elegant solution.

                    • Re: Connect Now Topology Discovery Problems
                      jstephenson

                      I am having this same issue, some (very few) of my devices were able to discover link info with Network sonar.  Upon checking, the OID's Solarwinds uses for this I find the are not being populated on the switches not working.  Does anyone know why on devices of the same make and model, same IOS, and using the same template for configuration these OID's are not getting populated?  Anyone know how to get them to populate??

                      Thanks!

                    • Re: Connect Now Topology Discovery Problems
                      Mr. Wick

                      Thanks to Will and Brandon for working with us on this.

                      We've been having issues with devices where single devices with multiple IPs will be discovered multiple times, but each will show different topology information. 

                      Also we've had instances where directly connected devices will show in CDP, but not in topology.  For some reason the switch port mapper isn't getting the MAC address for the connected neighbor.

                      Will, per our discussion this morning, there have been no IPS blocks between NPM server and devices at all.

                      I like this enhancement, and find great value in it's ability to quickly see connections.  Thanks for bringing it to us, and keep at it!

                      Dan

                        • Re: Connect Now Topology Discovery Problems
                          netlogix

                          *(if you have a Cisco switch and are using vlans)*

                          try making you SNMP string public@101 when you do the walk

                          (replace public with your snmp string and 101 with the vlan that device is on)

                            • Re: Connect Now Topology Discovery Problems
                              ecklerwr1

                              commstring@101 is interesting...

                              one thing I've noticed since running network sonar.... all my 3750's with shortened comm strings now fail list resources from the ManageNodes part of the web interface.  I can poll them... do rediscovery... and stats are still being collected but when I try and list resources or do edit and try and test my known good comm string... it fails.  I don't want to have to delete and re-add the cisco 3750 nodes to fix this.  When I run system manager and do a list resources is works perfectly like it always used to.  This started happening with 3750's after running network sonar to try and get topology info into the database.

                              I seem to remember other people having something like this happen possibly with 9.5 versions...

                              Is it safe to run configuration wizard on my production database and website without screwing up what's currently in NPM?  I was thinking there's a chance this might fix the Cisco 3750 switch problem.

                              Also NPM is currently and has been successfully collecting data from all the 3750's since running network sonar... it's just web interface managenodes list resources or edit and test snmp that fails... I even tried changing the comm string to my old long comm strings that never worked on my 3750's and then back to my shortened on to see if I could get it work again but.... no go :(

                                • Re: Connect Now Topology Discovery Problems

                                  ecklerwr1,

                                  You should be good to run the config wizard, but to be safe, be sure to grab a backup of your db before you do.

                                  W

                                    • Re: Connect Now Topology Discovery Problems
                                      ecklerwr1

                                      Thanks Will-

                                      I reran config wizard after backing up database and website on the database, website, and NPM Service.  As you suggested Will is didn't screw anything up.  Still can't get list resources to work with any of my 3750's... I ran workspace studio from the same machine NPM is running on and can poll the 3750's with same community string with out issue.

                                      I have no idea what it did but network sonar changed something in the database which now prevents list resources or testing snmp on the 3750's.  I tried saving the community string as something else then reverting back to the one I know works and list resources and edit -> test snmp with string still fails... all despite that NPM is correctly polling the 3750's still with snmp without issue... go figure :}

                                      At least this isn't keeping me from monitoring my 3750's... for the time being I have make monitored interface changes in system manager instead of from the web with ManageNodes for these switches.

                                        • Re: Connect Now Topology Discovery Problems
                                          Karlo.Zatylny

                                          I would sniff the network with wireshark and decode the SNMP packets and see if there is a difference in the community strings being sent.  Maybe a space snuck in there.

                                          Run the wireshark and do a poll now and a list resources on the same node and see what the difference in the packets is.

                                          Let me know if you need any help navigating SNMP on wireshark.

                                            • Re: Connect Now Topology Discovery Problems
                                              ecklerwr1

                                              @Karlo how could there be a space or something in there if NPM is successfully polling the device with the string that was in there working all along before network sonar ran?  I'll dig deeper and I have a case open 156592.  Here are the errors doing list resources or edit test snmp, and the same nodes current interface status gather with the snmp is says doesn't work:

                                              Seems like the same comm. string that fails the test and list resources is working fine for the current polling.  Everything worked fine and I hadn't changed anything from before... except running Network Sonar.

                                                • Re: Connect Now Topology Discovery Problems
                                                  Karlo.Zatylny

                                                  B. Eckler,

                                                  I am not sure why your list resources isn't working.  I was hoping to give you a method to start your investigation.  Using wireshark you can look at the bytes and the decoded SNMP to see what the difference is in the SNMP packets being sent. 

                                                  If they look identical there, then it is time to go to the device and see if the packet is making it there and if the device has SNMP logging, why it is rejecting the packet.

                                                  For a Cisco device:

                                                         

                                                  1. Login to the Cisco device

                                                  2. Enter the config mode by typing config t

                                                  3. Type logging monitor debugging

                                                  4. Exit the config mode by typing exit

                                                  5. Type debug snmp packets

                                                  6. Type terminal monitor

                                                  Start to view the output of the SNMP received by the device.

                                                  Thanks

                                    • Re: Connect Now Topology Discovery Problems
                                      Capt. Obvious

                                      This could be due to an '@' symbol in your community string.

                                       

                                      On cisco devices the @ symbol indicates 'Community string indexing' and this can wreck havoc on certain MIBs (such as the bridge-mib we use to detect topology)

                                      Community string indexing allows you to use the syntax : community@vlan# (i.e. public@40) to specify what vlan to query.

                                      So in turn iof you have a comminity string of p@ssp0rt it would fail as it wouldtake 'p' to be the community string and 'ssp0' to be the vlan detail and drop the 'rt' resulting in a failure.

                                       

                                      Not saying this is the issue but it definatly COULD be if you have an '@' in your community string.

                                        • Re: Connect Now Topology Discovery Problems
                                          ecklerwr1

                                          @Captain Obvious may have hit the nail on the head... interesting thing is in both my long comm string used on all my cisco devices except 3705's and my shortened version I have been using on 3750's without issue up to version 10 of NPM... they both have "@" symbols... the long one has two and the short string has one.  I'm coming to the conclusion that now with version 10 of NPM SolarWinds has changed something with the handling of community strings when using the "@" symbol that wasn't done before.  I've come to this conclusion because I haven't changed any of my devices (hardware or IOS version) or their community strings.  Now since v10 my long comm string with two "@" symbols still works but my shortened version with an "@" symbol in the 3rd character position fails to check in edit or list resources yet is still being successfully polled by NPM poller.

                                          It looks like I need to work on changing all my community strings after doing a couple first as a test to see if this resolves the issue.

                                          @Karlo - I'm investigating the connect now issue further.

                                          • Re: Connect Now Topology Discovery Problems
                                            ecklerwr1

                                            I have verified that Captain Obvious was correct about running into issues with "@" in my community strings on 3750 Cisco switches.  I changed the "@" symbol in my community strings to "#" and I can now list resources and pass the snmp test in edit nodes on my 3750's.  Props go out to Capt. Obvious and Karlo for helping me work this out :}

                                          • Re: Connect Now Topology Discovery Problems
                                            smartd

                                            One thing I figured out:

                                            When you run a sonar discovery, it needs to complete to the end as this is when topology gets discovered.  I had to up the timeout to 240 on big /16 subnets.

                                            Your topology data is linked to the discovery "session".  If you rerun the session, the new topology overwrites the old one.  If you reuse a discovery session/config and change the IP addresses scanned, the old topology data from the previous IP range WILL BE DELETED. 

                                            Routers seem to need LLDP or CDP to work.

                                            Note: All these conclusions are empirical.

                                            -=Dan=-

                                              • Re: Connect Now Topology Discovery Problems
                                                Questionario

                                                sounds like this connect now is still very buggy...

                                                I hope it won't be in the next release and instead will be released bug-free in the next release...

                                                • Re: Connect Now Topology Discovery Problems
                                                  ecklerwr1

                                                  @Dan

                                                  So you are saying if you edit an old saved discovery session and change the subnet that it would delete the original subnets topology information from the the first time you ran the session?  This is strange behavior.  I have some other issues with comm. string also but thought the topology could be learned multiple ways including the bridge mib... it seems my routers don't have the two OIDs listed above populated and due to security requirements cdp is disabled on all of our routers... seeing how you mentioned routers seem to need cdp it may be the reason I'm not getting topology data populated...

                                                    • Re: Connect Now Topology Discovery Problems
                                                      netlogix

                                                      I have CDP disabled on all my routers and switches but I still get some topology data.  I have two Cisco 3560 switches that won't map out ports but many others that do.  The IOS version is the same and the configs are basically the same.

                                                      Have you tried Solarwind's LANsurveyor? There is a 21 day (<- why not 30!) free trial that you can try.

                                                      • Re: Connect Now Topology Discovery Problems
                                                        smartd

                                                        @ecklerwr1

                                                        This is all from running tests and seeing what happens.  I kept having topology data disappear.  I finally figured out that it was happening because I was using a SINGLE discovery "rule" or "discovery" or whatever you call it.  I'd run it with one subnet and import the results.  Then I'd edit the discovery rule and change the subnet and run it again.  The topology data from the previous run was deleted and new topology was created.  When I ran it again with the first subnet, the topology data swapped back. 

                                                        My inference is that topology data is tied to the interface, the discovery "rule", and the date the rule is run.  If you remove an interface using Node Details/List Resources, the topology data is deleted for that interface.  I'm not sure what happens if you delete a discovery results

                                                        After running scheduled discoveries for a week, and trying to develop a process for importing results, I see this may be too resource intensive in its present form.  To make the process more manageable, I'd like to see the following:

                                                        1. I'd like the scheduled discoveries to have an option of ONLY reporting on, and importing, interfaces that HAVE topology data AND are discoverable via SNMP-only.  This gets all the trunks and uplinks and important gear with communities that the NOC manages.

                                                        2. I'd like to have the ability to add an interface to the  ignore interfaces from discovery list from the Node Details/List Resources screen! 

                                                        3. I'd like the ability to ignore a node from discovery AND be able to delete a node directly from the Node Details "Node Details Resource"

                                                        4. I'd like the ability to ignore an interface from discovery AND be able to delete an interface directly from the Interface Details / "Interface Details Resource"

                                                        5. I'd like a flag on the scheduled discovery results screen that shows that devices have already been imported. (Or have them disappear after importing).

                                                          • Re: Connect Now Topology Discovery Problems
                                                            Karlo.Zatylny

                                                            smartd is hitting close to the way topology data is imported.

                                                            The topology data is updated (read completely deleted and rewritten) for each execution of every individual discovery profile.  So updating the IP addresses in a single discovery profile, will overwrite/delete the data from the previous discovery.

                                                            The intended use is to have several separate discoveries that each poll a certain set or range of IP addresses.  Then each discovery profile is responsible for updating its set of the topology data.  Reusing the same profile will keep overwriting your data.

                                                            Let me know if I can clarify anything. 

                                                            And as for your suggestions... I am sure the PMs will take note of these.

                                                            Thanks

                                                              • Re: Connect Now Topology Discovery Problems
                                                                smartd

                                                                Thanks for the confirmation Karlo,

                                                                1. If you delete the results from the scheduled discovery results page, does topology get deleted from the database?

                                                                2. If you delete a discovery profile, does the topology data already discovered get deleted?

                                                                  • Re: Connect Now Topology Discovery Problems
                                                                    chrkov

                                                                    Is it planned to have Layer 3 devices work with connectnow??  CDP would have all the needed information.

                                                                    • Re: Connect Now Topology Discovery Problems
                                                                      ondrej.salplachta

                                                                      Thanks for the confirmation Karlo,

                                                                      1. If you delete the results from the scheduled discovery results page, does topology get deleted from the database?

                                                                      2. If you delete a discovery profile, does the topology data already discovered get deleted?

                                                                      1. What do you exactly mean by delete results from the scheduled discovery results page? There is not possible to delete discovery results. There is possible to add nodes into ignore list, import them or run discovery again. When you add node into ignored list, it will be still in the topology. Topology could be created only for imported (managed) nodes. When you run discovery again, topology for this profile is replaced by the new results.

                                                                      2. If you delete discovery profile, everything for this discovery profile is deleted: Profile definition, Discovery results and Topology data.

                                                                        • Re: Connect Now Topology Discovery Problems
                                                                          smartd

                                                                          Ondrej

                                                                          1. You're right.  You can't delete. My mistake.

                                                                          On the ignore list...  So if you move a device to the ignore list, that makes the topology "frozen".  If you run a discovery profile again, will ignored device topology be updated?

                                                                          Thanks

                                                                            • Re: Connect Now Topology Discovery Problems
                                                                              ondrej.salplachta

                                                                              When you add node to the ignored list and run discovery again, topology is still updated for this node. Otherwise, ignore list hasn't impact to topology.

                                                                                • Re: Connect Now Topology Discovery Problems
                                                                                  smartd

                                                                                  Thanks Ondrej:

                                                                                  So here's my understanding of the Network Discovery activity:

                                                                                  1. One Discovery profile is created for each subnet you intend to acquire topology and device updates.  These are scheduled to run periodically. 

                                                                                  2. If the discovery is actively monitored, a dialog for importing results is given AND results are listed on Scheduled Discovery Results.

                                                                                  3. If the discovery is scheduled, the results are found on the Scheduled Discovery Results and a banner alert is given.

                                                                                  4. The Scheduled Discovery Results shows all devices discovered by all Discovery Profiles that have been run.  The results stay there until one of three things happens:

                                                                                  - The Discovery Profile is deleted, which deletes all topology data and discovery results
                                                                                  - The Discovery Profile is rerun, which updates all topology data and discovery results
                                                                                  - Or devices are sent to the Ignore list, which makes them disappear from the Discovery Results screen but are still part of the Discovery Profile and will be updated or deleted if the profile is rerun or deleted.

                                                                                  5. Importing (to NPM)  has no affect on the Scheduled Discovery Results.  This acts a a repository of devices you can choose to manage or NOT manage in NPM.

                                                                                  ==================================================

                                                                                  Do I have this correct?

                                                                                  When the Scheduled Discovery Results shows a node or interface as "new" or "changed".  Is this in relation to the previous Discovery, or new to managed devices in NPM?

                                                                                  Also, what triggers the Banner message on new devices?  Again, new to NPM or new to Discovery?

                                                                                  -=Dan=-

                                                                                    • Re: Connect Now Topology Discovery Problems
                                                                                      ondrej.salplachta

                                                                                      1. Correct

                                                                                      2. Results are listed on Scheduled Discovery Results (SDR) only for profiles which are scheduled (not for results from manual profiles). When the discovery progress is shown and discovery finish, page with "Import discovery results" is shown (not SDR).

                                                                                      3. Correct

                                                                                      4. Correct

                                                                                      5. When you import nodes to NPM, the node in SDR will be marked as: "Imported" (in case you imported all interfaces and volumes) or "Changed" (in case some Interface or Volume isn't imported).

                                                                                       

                                                                                      Node can be changed from "Imported" to "Changed" in these cases:

                                                                                      1. During discovery is found another new Interface or Volume for this Node

                                                                                      2. Some already imported Interface or Volume is deleted in Node Management (managed nodes)

                                                                                      Summary about status in SDR:

                                                                                         "New" status is there when node isn't imported at all (Nodes are compared by Engine ID and IP address).

                                                                                         "Changed" is shown when node is already imported but in SDR are some Interfaces or Volumes which are not imported yet.

                                                                                       

                                                                                      The banner is shown only for Scheduled Discovery when discovery finish and in results are some "New" or "Changed" nodes.

                                                                                        • Re: Connect Now Topology Discovery Problems
                                                                                          smartd

                                                                                          @ondrej

                                                                                          Yes, "found" devices disappear after doing an import.

                                                                                          No, "changed" devices do not disappear after doing an import, as interfaces it finds are not UP so don't get imported.  It stays in the "changed" list forever.

                                                                                          I have found that getting interfaces and nodes into the ignore list is the most important process for scheduled discovery sanity.  There are very few features to help in this process.  I can't expand all, i can't list all the interfaces the way node manager will, I can't search for certain interfaces like loopbacks.  I can't set a rule like "ignore all loopbacks on HP Procurve switches".  Got to ignore them ONE AT A TIME.

                                                                                          What I'd like is to have a results view that just shows interfaces with a status of UP and not monitored in NPM (not all interfaces not in NPM), and all interfaces with topology data not monitored in NPM.  There is a 100% chance I want to manage topology related interfaces.  New "UP" interfaces may or maynot be managed depending on the device. 

                                                                                          I'd also like a way to update custom fields in import.  Or at the very least, and shortcut to Node Manager to update them.

                                                                                      • Re: Connect Now Topology Discovery Problems
                                                                                        smartd

                                                                                        Ok, I went through all my "found" devices in Scheduled Discovery Results. 

                                                                                        Many of my Cisco routers with an IP of .254 and a subnet of /24 were included in the found list with BOTH a .0 and .255 address.  These are reserved addresses in the subnet, so something is amuck with discovery and subnets.

                                                                                        I moved all these .0 and .255 addresses to "Ignore". 

                                                                                        I did an import on what was left.  The status of these devices did not change.  Still "New"

                                                                                        I looked at all the filtering options for this results screen and see one of them is "imported".  What I really need is one called "not-imported" so I can see the ones I still need to deal with.  

                                                                                        There is also a bug with Juniper (JUNOS) Proprietary Interfaces.  Each Juniper interface has a physical interface and one or more logical interfaces.  When I do an import, and request that only interfaces that are "up" are imported, ALL of the proprietary interfaces are imported... both down and up. 

                                                                                        I'd also like an option during import to import all interfaces with topology data.  That way you could JUST monitor "point to point" interfaces and not edge interfaces.

                                                                                        -=Dan=-

                                                                              • Re: Connect Now Topology Discovery Problems
                                                                                ecklerwr1

                                                                                This is what I intuitively did and seems to cause the topology to be deleted as you found:

                                                                                Then I'd edit the discovery rule and change the subnet and run it again.

                                                                          • Re: Connect Now Topology Discovery Problems

                                                                            So is it recommended to have a discovery profile for each subnet? 

                                                                             

                                                                            1. For example if we use 172.16.0.0 /24 with 8 /24 subnets underneath, that I should make 8 discovery profiles? 

                                                                            2. Does ConnectNow topology change get reflected in a Network atlas map?  I didnt think so but wasnt sure.

                                                                            3. I still get some switches (same model, same IOS, same connectivity between switches) that dont get connected.  Not sure how to debug.  I can browse the bridge mib on all from the Orion server.

                                                                            I have tried a varied amount of discovery profiles in an attempt to get a decent topology layout.  I am a bit restricted as we have the Cisco Nexus switches which I learned from other threads, does not support the bridge mib.

                                                                             

                                                                            Thanks

                                                                              • Re: Connect Now Topology Discovery Problems
                                                                                smartd

                                                                                inrouted:

                                                                                I'm assuming you mean 172.16.0.0/16.  We find that a/16 mask discovery takes about 4 hours to complete, so we set the timeout to 300 minutes.

                                                                                By dividing the supernet up into smaller subnets, the each scan would complete sooner of course. 

                                                                                One bug I've found is that scanning the whole /16 subnet, sometimes the broadcast addressess of the real subnets will get picked up as nodes.  I always check the discovery results to make sure I don't have any .0 or .255 routers show up.  (Note: this is not an oddball device issue, it happens with our 1841 Cisco routers).

                                                                                -=Dan=-

                                                                                  • Re: Connect Now Topology Discovery Problems

                                                                                    Ahh yes.  I did mean /16.  Brain was thinking ahead to later in the  sentence.  So did you make multiple discoveries and increase the hop  count?  or do you have 1 discovery which you then add nodes to the  ignore list.  Too bad it doesnt correlate cdp information as well as the bridge mib.