23 Replies Latest reply on Jun 21, 2013 12:55 PM by deverts

    SNMP Get Requests on Firewall

    Scott Sadlocha

      Hey All,

      I had a question thrown at me today by our firewall administrator, but I can't narrow down the answer, so I am looking for help. He has noticed SNMP get requests on our Cisco ASA devices from something in the Solarwinds environment to IP addresses in the 192.168.0.1-5 range. We are currently not using these addresses, so I am not sure why they are being queried, but it seems to be occurring daily. This is triggering on our IPS so there is some concern, and he would like to reduce our signal to noise ratio. Based on the IP address and the fact that we don't use them, I am figuring this is something running with a default value, and I suspect it might have something to do with the Engineer's Toolset. Since he is the person that installed Solarwinds in our environment, I asked him about the toolset, and he said that he previously had it installed, but said he no longer did and he wasn't aware of anyone else having it.

       

      However, I found that browser integration is turned on in the web console, and the toolset install prompt is there, so I wonder if that can have something to do with this activity. If anyone can provide any information on a possible cause, I would appreciate it.

        • Re: SNMP Get Requests on Firewall
          Scott Sadlocha

          Just wondering if anyone has had a chance to give this much thought yet. I am hoping someone has seen this before and can offer some input.

          • Re: SNMP Get Requests on Firewall
            mltownsend

            Are you using any other Solarwinds products? I had a similar issue, and the culprit was IPAM scanning the entire subnet.

            • Re: SNMP Get Requests on Firewall
              Scott Sadlocha

              Okay, I have been looking around in settings a bit. While IPAM seemed like it could fit, I could find no listing of the IP addresses I mentioned anywhere in IPAM. Not in the addresses, scope or scan settings.

               

              I then started going to other applications, starting with NTM, another application I am not very familiar with. I checked the Edit IP Address Groups settings in NTM, and I found "Number of Selected Groups: 3" with the Enabled checkbox checked for group Private Addresses 192.168.0.0 - 192.168.255.255.

               

              The information on this screen states the information below. Do you think that this could be causing the requests? I wouldn't think that this would result in an SNMP request, but so far it is the only instance I have found of the IP addresses in question.

               

              Check groups to enable monitoring. Checked groups will appear in the Top XX IP Address Group resource.

              IP Address groups can also be used to define application in Manage Applications and Service Ports.

               

                • Re: SNMP Get Requests on Firewall
                  deverts


                  Scott,

                   

                  NTA doesn't perform any network discovery to my knowledge, that setting is there so it knows how to classify the data it is receiving.  Additionally, SAM doesn't actually do any network discovery either.  It scans systems for applications when it is manually invoked, but to my knowledge is also not automatic.

                   

                  The likely culprits are NPM, specifically a scheduled Network Sonar discovery, or IPAM scanning.  Are the scans being detected on a schedule?  Default for IPAM is every 4 hours, and typically NPM scans are daily (but that really depends on the person that set it up).

                   

                  D

                    • Re: SNMP Get Requests on Firewall
                      Scott Sadlocha

                      Thanks for the additional information on NTA and SAM. I figured that might be the case but wanted to mention it. I suspected NPM with the discovery and checked settings on that, but the settings indicate only our available networks, and no reference to the unused IP addresses I referred to.

                       

                      So right now it looks like IPAM is the most likely candidate, most likely with the Neighbor Scanning option that mltownsend mentioned. Our firewall guy had to work on something last night and isn't in yet, but when he gets here I plan to get more details from him, including timing of the events he noticed. Perhaps then I could disable that option and see what kind of results we get. I will report back here once I get that done.

                       

                      Going back to my original post, do you think there might be something in the toolset that could be doing this, if it is still installed somewhere?

                        • Re: SNMP Get Requests on Firewall
                          deverts


                          Yes to all the above.  IPAM does sound like a likely candidate; and if the toolkit is installed, that could be it also depending on how someone has it configured.  As a network and network security engineer myself, your IPS and/or firewall should be providing you with the IP address of the source.  Did your firewall guy provide the source IP of the SNMPGET?  You mentioned it was coming from your SW/Orion environment, so someone knows the actual source IP.

                           

                          D

                    • Re: SNMP Get Requests on Firewall
                      Scott Sadlocha

                      Okay, I got a bit more info. I was incorrect about the addresses being scanned. They are actually 192.168.2.x addresses, and the scan seems to be targeting addresses .3 through .8. Either way, the addresses are not being used in our environment. I actually looked at the firewall log with our admin, and saw the listing for the events. It is happening at just before 12:30AM Eastern daily (12:26AM in the latest instance) and it is always the same type of events. It is showing as an SNMP Community String Brute Force Request which is most likely and SNMPGET request, and it always comes from the IP address of our Orion server.

                       

                      I just checked IPAM, and the scan settings are still at the default, every 4 hours. I figured I might as well check directly on the server. Thinking it might have something to do with Traps, I checked Trap Viewer. While I don't see those IP addresses in there, I do see a bunch of traps at the same time. These traps are in regard to one of our IIS servers and there looks to be authentication failure. Listed in the trap entry is this for the trap type: SNMPv2-MIB:authenticationFailure. Also showing up under Trap Details (the blank line is information on the server, including IP address):

                      snmpTrapOID=SNMPv2-MIB:authenticationFailure

                      sysUpTime=55 days 12 hours 59 minutes 27.74 seconds

                       

                      snmpTrapEnterprise=MSFT-MIB:server

                       

                      Does this shed any light on anything familiar for anyone, perhaps give guidance for somewhere else I might look? Could the traps connected to the IIS server be generating the requests on the additional IPs?

                        • Re: SNMP Get Requests on Firewall
                          deverts

                          Scott,

                           

                          If you open the node details for that IIS server, in the Node Details resource you will see a "Next Discovery" date/time.  Does it read something like "6/14/2013 12:25PM?"  The default behavior for NPM is to rediscover each node on a daily basis, and that rediscovery time is initially based on the date/time it was added to NPM.  I think this is want you are seeing.  As for the authentication error, I would "Edit Node" and reapply the SNMP string to the server, and "Rediscover" the node.  If that doesn't fix it, get the SAs to look at the SNMP security settings on the server, they might be preventing your Orion server the SNMP access it needs for a full data pull.  The firewall log is probably seeing this as an attack because NPM is trying multiple times to get the data.

                           

                          Just my thought...let us know.

                          D

                            • Re: SNMP Get Requests on Firewall
                              Scott Sadlocha

                              I just checked that server, and it is being monitored via WMI and ICMP, which surprised me. A service account is being used for the polling. I can see drive and volume resources for the server, but of course no interfaces since SNMP is not set up. The Next Rediscovery is set for 2:22PM, a few minutes from now. The rediscovery interval is set to 90 minutes. Looking at this server, I can't see anything that really corresponds to those entries. Do you think it is a coincidence seeing those traps entries at the same time?

                                • Re: SNMP Get Requests on Firewall
                                  deverts

                                  I don't believe coincidence when it comes to IT.    To many moving parts and everything impacts everything somehow.  Your findings sort of push me back towards the SNMP discovery process in IPAM.  IPAM tries to use SNMP to pull general system info from devices it located with ICMP.  I don't think you can disable the SNMP polling on a per subnet basis either, I think its all or nothing.

                                   

                                  And I think we are chasing our tails here, so here's some steps to help isolate where the issue is:

                                   

                                  1.  (eliminate the network)  If this is a result of the IPAM neighbor scanning, then some router will have ARP entries for this network in it.  Check the ARP tables of the routers, I'd start with the core if you can and work your way out. (don't be surprised if you locate a DHCP-enabled mini-hub)

                                  2.  (eliminate IPAM)  In the IPAM search window, type in "192.168.2*" and search for that subnet...if it exists, what else is in the subnet?

                                  3.  (eliminate NPM)  Go to the Managed Nodes list and look for something with an IP in the range 192.168.2.x

                                   

                                  If you come up negative on all those, then you RDP into the Orion server, open the database manager, and start looking for the needle in the haystack.

                              • Re: SNMP Get Requests on Firewall
                                deverts


                                Additional info that I just thought about...what are the results if you "List Resources" on that IIS server?  Do you see the disks, memory and NICs?  That would also validate your SNMP strings.

                                 

                                D

                              • Re: SNMP Get Requests on Firewall
                                Scott Sadlocha

                                Okay, I think I am getting closer to figuring this out. I figured that I would have to start digging to find the problem. Unfortunately I had trouble accessing the DB with the rights I have, and I have been waiting for someone to provide the correct rights. However, yesterday I decided to run diagnostics and start digging through the output. Since Autodiscovery ran at 12AM every day, I figured I would start with that. As I looked through the OrionDiscoveryJob log file, I started finding some interesting entries, including some related to the IP addresses I previously mentioned. It looks like autodiscovery is pulling the information from our VMs somehow, but I am not sure what to make of it. For each address, the entries below show up. They are not all consecutive, but spread out throughout the second half of the log file, and I put spaces in between the entries that occur in different sections. Usually they show up one address after the other in each section. Again, if I search for these addresses, they don't show up in Orion, and we are not using these addresses as far as I know in our environment.

                                 

                                I am going to try to suspend autodiscovery for a day to see what happens, but it looks like you can't actually just say "Suspend" but rather have to change the schedule. Looking through the Network Sonar Wizard that comes up when I click on the job to Edit, I noticed a couple things and wonder if they could help with the issue (just going through each step in the wizard here). We have 6 SNMP community strings set up in the scan, and it looks like we have 4 custom along with the usual "public" and "private".  We also have the "Poll for VMware" box checked and 2 sets of vCenter/ESX credentials defined. We also have a single Windows service account set up. We have 4 subnets defined, none of which contain the offending addresses. On the Discovery Settings slider page, we have SNMP and WMI Retries both set to 1, Hop Count set to 0. Interestingly, on this page, the checkbox for "Ignore nodes that only respond to ICMP (ping). Nodes must respond to SNMP, WMI." is checked. The two items I wonder about with regard to the issues I am seeing are the "Poll for VMware" and the "Ignore nodes..." options. I wonder if either of these could be causing autodiscovery to look for something that isn't there after finding something in one of our VMs.

                                 

                                00:23:04,8 [STP SmartThreadPool Thread #0] DEBUG VimDuplicationMerge - Deduplication merge of IP: 192.168.2.4  into IP: 192.168.2.4  because of identical address.

                                00:23:04,8 [STP SmartThreadPool Thread #0] DEBUG DiscoveryProcessResultManager - Found duplicate against another generated item: 192.168.2.4.

                                 

                                00:23:05,1 [STP SmartThreadPool Thread #0] DEBUG DetectionPhase - Next endpoint for processing: IP: 192.168.2.4

                                00:23:05,1 [STP SmartThreadPool Thread #0] DEBUG DetectionPhase - Start waiting for next task.

                                00:23:05,1 [STP SmartThreadPool Thread #0] DEBUG DetectionPhase - Endpoint IP: 192.168.2.4  will be processed.

                                 

                                00:24:35,4 [STP SmartThreadPool Thread #68] DEBUG WmiHelper - CreateWmiRequest Request:192.168.2.4\root\CIMV2 [Shoremortgage\SolarWindsSvc!], Query: [select Manufacturer,Version from Win32_OperatingSystem], Options: [LI], Retries: 1, RetryInterval: 10000

                                 

                                00:25:01,0 [STP SmartThreadPool Thread #68] DEBUG WmiConnectionBuilder - WMI connection attempt to \\192.168.2.4\root\CIMV2 failed. Attempt 0 of 1

                                00:25:01,1 [STP SmartThreadPool Thread #54] DEBUG WmiConnectionBuilder - WMI - unable to resolve target computer host name or ip address.

                                00:25:01,1 [STP SmartThreadPool Thread #22] DEBUG WmiConnectionBuilder - WMI - unable to resolve target computer host name or ip address.

                                 

                                00:25:36,6 [STP SmartThreadPool Thread #68] DEBUG WmiConnectionBuilder - WMI connection attempt to \\192.168.2.4\root\CIMV2 failed. Attempt 1 of 1

                                 

                                00:25:41,1 [STP SmartThreadPool Thread #68] DEBUG WmiProbe - WMI failed on 192.168.2.4: Unresponsive: The RPC server is unavailable. (Exception from HRESULT: 0x800706BA)

                                00:25:41,1 [STP SmartThreadPool Thread #23] DEBUG WmiConnectionBuilder - WMI - unable to resolve target computer host name or ip address.

                                 

                                00:26:02,2 [STP SmartThreadPool Thread #10] DEBUG VimDetection - VIM detection phase - probe start.

                                00:26:02,2 [STP SmartThreadPool Thread #10] DEBUG VimDetection - VIM detection phase end.

                                00:26:02,2 [STP SmartThreadPool Thread #68] WARN  VMwareProbe - Cannot connect to a VMware node on https://192.168.2.4/sdk

                                00:26:02,2 [STP SmartThreadPool Thread #68] DEBUG VimDetection - VIM detection phase - probe failed on 192.168.2.4.

                                 

                                00:26:19,6 [STP SmartThreadPool Thread #0] DEBUG DiscoveryProcess - Duplicates for endpoint IP: 192.168.2.4 :

                                00:26:19,6 [STP SmartThreadPool Thread #0] DEBUG DiscoveryProcess -  Endpoint: IP: 192.168.2.4  Reason: Identical IP addresses

                                  • Re: SNMP Get Requests on Firewall
                                    Scott Sadlocha

                                    I would like to figure out how to clear these entries, but can't see how I would do so in Solarwinds. Would I add the addresses to an Ignore list? It seems like this is something that needs to be cleared up within the VM environment.

                                    • Re: SNMP Get Requests on Firewall
                                      deverts

                                      Scott,

                                       

                                      That's perfect info, thanks!  I suspect what you are seeing is the result of the SNMP poll of the VMs.  Part of the polling process is to pull the ARP cache from the device it is polling, and then try to poll those devices (the "neighbors").  I'm assuming your VM environment is using 192.168.2.x as its VM interconnect network.  This is typically a non-routable, private network that doesn't leave the VMs, but since NPM is seeing the ARP entries, it is trying to poll those devices.

                                       

                                      So, you are correct, the ignore list is the way to go.

                                       

                                      D

                                        • Re: SNMP Get Requests on Firewall
                                          Scott Sadlocha

                                          Thanks so much for the explanation. I was hoping the Ignore List would be the way to go, but it doesn't look like that will work for me. It seems that I can't manually add addresses to it unless I do it on a database level, which is not an option right now. Nodes need to be found via discovery and show up in the results to be added to the Ignore List, but these addresses don't even show up in results. I am guessing that unchecking the "Poll for VMware" or the "Ignore nodes..." checkboxes I mentioned in my previous post might accomplish the task (especially the "Poll for VMware" option), but I also wonder if doing this will also exclude other items we might want to find from the discovery.

                                      • Re: SNMP Get Requests on Firewall
                                        Scott Sadlocha

                                        Just wanted to say thanks. I ran a diagnostic and started looking through the logs. I found multiple entries in the Network Sonar Discovery logs for the IPs I mentioned. Many were in context to discovery scans of the VMs. I then disabled the discovery scan for a night and the entries did not show up, so that confirmed it a bit more. I then started looking into our VM environment and found that there was an old VM for a file server cluster that is no longer used and is scheduled to be decommissioned in the next couple weeks. The VM was using those IP addresses as its internal addresses as you mentioned.