How do you resolve missing mac addresses?

We have some specific IoT devices that are pretty well locked down. To the point where we get pretty much no information from them in IPAM, other than the fact that the IP address is in use. We would really like to get mac addresses from these devices, and switchport details would be nice as well. 

We do have UDT as well as IPAM, and naturally UDT has the opposite problem; we scan the switch to which these devices connect and we of course can see the switchport and mac addresses, but we get no IP addresses in UDT. It is a layer 3 switch, and we get mac and IP address information fine from other types of devices without any issue. I have done some SNMP queries, and have confirmed that the arp table is quite accessible via the OIDs these products use for scanning. It would just be really nice if the two products were able to correlate this data. 

I have turned on neighbor scanning for the affected subnet (fortunately, it's mostly all within a single subnet), but that brings its own issues. IPAM does indeed retrieve the mac addresses through neighbor scanning, but the next subnet scan overwrites them so they aren't readily available (other than via history). We have both automatic scanning and neighbor scanning enabled; should these be mutually exclusive? The documentation is, of course, quite sparse on these kinds of details.

  

The department responsible for these devices would really like to get the mac addresses to aid with identifying and tracking these devices. So, I guess I have a couple of questions for the community:

1. Is there any way to correlate mac addresses and IP addresses between UDT and IPAM? The information is all there; we just need a way to bring them together, preferably in both products? If not, does this seem like a reasonable feature request?

2. Neighbor scanning seems like a potential solution here, but should automatic subnet scanning be disabled for it to be useful? If so, this really needs to be documented. In this case, it may be reasonable to ONLY use neighbor scanning, but I could see this being problematic for subnets with a wider range of devices.

Thoughts? Just trying desperately to come up with some solutions for this quandry, and hopefully gain some further understanding of how these products should work.

Thanks.

Parents
  • The simple answer is you need to frame your mind that UDT and IPAM can't actually source data from each other.  There is only the most minimal amount of crossover.  Makes more sense when you consider that they were both developed entirely separately so the devs couldn't assume that someone had both modules until someone in marketing decided to start pitching them as a bundle and then ultimately rolled them into the new licenses.   The way they are marketed and sold does not reflect the way they were built.  I think it's a good feature request for them to integrate the two much better, and with the way they are licensed now it really doesnt make sense for them to remain separate anymore, but thats also kind of a big project in terms of how both of them work at this time.  At the last company I had where we had both products I did some pretty heavy editing of the UDT endpoint views and got them where they would more clearly cross reference against all my other modules, but I don't have a copy of that handy to share anymore.  It was just a bunch of Custom Query widgets combined with me spending a lot of time crawling through the db tables piecing together whatever I could find that looked useful.

    So getting to your number 2, yes neighbor scanning should help assuming all the pieces come together.  Neighbor scanning and automatic scanning arent really related.  Automatic scanning controls if/when the poller does a ping swing through the defined IP's and tracks if it gets responses or not.  Some people dont want all those ping sweeps all across their network, or they are documenting subnets that the poller wouldn't be able to reach so they disable it.

    Without neighbor scanning IPAM is hoping it has SNMP access to your device. Frustratingly related to that lack of integration I mentioned this scan has to happen within IPAM, which has its own collection of SNMP creds that it can't just inherit from all the other places where you would have already defined them.  IPAM wants to do l2 and l3 discovery against every IP, even though NPM and UDT are also doing those exact same SNMP requests as well and writing the data to different tables. Neighbor scanning kicks in when you can't SNMP poll the device for some reason, but you have to specify the IP of a device that would have the mac address in the arp table.  Given that UDT works for these macs you know you can get this data, its just going to be a matter of making sure your config is all correct inside IPAM for it to put it all together.

    https://support.solarwinds.com/SuccessCenter/s/article/How-IPAM-gets-a-MAC-Address-via-SNMP?language=en_US

    https://documentation.solarwinds.com/en/success_center/ipam/content/ipam-neighbor-scanning.htm

    Getting into the land of just hacking things up, in my past environment when I had discrepancies like this I once disabled all those kinds of topology polling in UDT and IPAM that all try the same OIDs and I used SQL queries to vacuum all the data I wanted out of the NPM topology tables and inserted it into the relevant tables in other modules.  Worked well for that environment and cut back on extra loads in my pollers and the target devices.

  • Thanks for your analysis. I have resorted to a lot of what you mention; e.g. routing through the tables to see what information I can extract from where. Part of my frustration comes from Solarwinds touting more integration between UDT and IPAM in the more recent releases, but this basic (in my mind, at least) functionality is still not present. And, yes; I do understand that these were different products, probably acquired and definitely developed separately. And I know that much of the work over the last three years is to work on getting all these different modules to share a common framework. But progress seems slow. I've been working with Solarwinds for a long time (15+ years?), and I'm not getting any younger. But I digress.

    As I looked into this further, and particularly looked at the example screenshots I added originally, I see that I was mistaken about the subnet scan being the source of the mac address removals. Instead, I see two distinct events happening; a 'Neighbor scan' and a 'Subnet neighbor scan.' The 'subnet neighbor scan' seems to be recording the mac addresses, while the 'neighbor scan' seems to be removing them. 

    The documentation (as we can all attest to) is frustratingly vague, but I assume these two items refer to the two different places one can enable neighbor scanning; e.g.

    • The "global" subnet scan settings, where the default appears to be that 'SNMP neighbor scanning' is enabled (i.e. Configure subnet scan settings manually). I'm assuming that this is the 'Neighbor scanning' associated with my disappearing mac addresses (based on very little data).
    • The "subnet" settings, where the default is to Disable Neighbor scanning (i.e. Neighbor scanning in IPAM). I assume this is the 'Subnet neighbor scanning' that is recording the mac addresses for me, however briefly.

    So, we have two different settings that appear to do similar things. But the one at the subnet level is configurable, allowing you to set intervals and the IP address of the device to scan, while the other is simply on or off. And it appears that one giveth, and the other taketh away.

    I know I'm not the first one to raise questions about how these two settings do (or do not) relate to one another (i.e. How does Neighbor Scanning work?), but I can't find any response that provides any clarification. So if anyone can shed some light on these two distinct (and possibly conflicting) settings, I'm sure we would all appreciate it.

    In the meantime, I guess I'll keep experimenting. I may disable the global 'SNMP neighbor scanning' next and see that that does. Maybe I'll actually be able to retain those mac addresses in this particular subnet, but I'm concerned that I may lose something elsewhere.

Reply
  • Thanks for your analysis. I have resorted to a lot of what you mention; e.g. routing through the tables to see what information I can extract from where. Part of my frustration comes from Solarwinds touting more integration between UDT and IPAM in the more recent releases, but this basic (in my mind, at least) functionality is still not present. And, yes; I do understand that these were different products, probably acquired and definitely developed separately. And I know that much of the work over the last three years is to work on getting all these different modules to share a common framework. But progress seems slow. I've been working with Solarwinds for a long time (15+ years?), and I'm not getting any younger. But I digress.

    As I looked into this further, and particularly looked at the example screenshots I added originally, I see that I was mistaken about the subnet scan being the source of the mac address removals. Instead, I see two distinct events happening; a 'Neighbor scan' and a 'Subnet neighbor scan.' The 'subnet neighbor scan' seems to be recording the mac addresses, while the 'neighbor scan' seems to be removing them. 

    The documentation (as we can all attest to) is frustratingly vague, but I assume these two items refer to the two different places one can enable neighbor scanning; e.g.

    • The "global" subnet scan settings, where the default appears to be that 'SNMP neighbor scanning' is enabled (i.e. Configure subnet scan settings manually). I'm assuming that this is the 'Neighbor scanning' associated with my disappearing mac addresses (based on very little data).
    • The "subnet" settings, where the default is to Disable Neighbor scanning (i.e. Neighbor scanning in IPAM). I assume this is the 'Subnet neighbor scanning' that is recording the mac addresses for me, however briefly.

    So, we have two different settings that appear to do similar things. But the one at the subnet level is configurable, allowing you to set intervals and the IP address of the device to scan, while the other is simply on or off. And it appears that one giveth, and the other taketh away.

    I know I'm not the first one to raise questions about how these two settings do (or do not) relate to one another (i.e. How does Neighbor Scanning work?), but I can't find any response that provides any clarification. So if anyone can shed some light on these two distinct (and possibly conflicting) settings, I'm sure we would all appreciate it.

    In the meantime, I guess I'll keep experimenting. I may disable the global 'SNMP neighbor scanning' next and see that that does. Maybe I'll actually be able to retain those mac addresses in this particular subnet, but I'm concerned that I may lose something elsewhere.

Children
  • So....

    FYI. The 'Enable SNMP neighbor scanning' under the subnet scan settings is basically an on/off switch for all subnet neighbor scans. Disabling that setting disabled the neighbor scanning for any subnet where I had unset 'Disable Neighbor scanning' (nice double-negative in this case).

    So, the essential logic is:

    • To enable neighbor scanning for a subnet, the global 'Enable SNMP neighbor scanning' setting must be enabled and the 'Disable Neighbor scanning' setting must be disabled at the subnet level.
    • To disable neighbor scanning for a subnet, you can either:
      • Leave the both the global and subnet settings enabled (essentially enable SNMP neighbor scanning globally, but Disable it for the subnet, which is the default)
        OR
      • Disable the global setting, which effectively tells IPAM to ignore where 'Disable Neighbor scanning' is disabled

    So, now we know.