we've seen similar case before, where it was caused by something between Orion and the polled device. It looked like the SNMP requests were being blocked or dropped by a firewall.
In the wireshark trace (run on the Orion server), we saw that the requests were being sent but no response was coming back.
Can you run wireshark on your server at the time the issue occurs to check if response is coming back for that particular node?
If it doesn't, can you restart the firewall to see if it makes any difference?
That's exactly what we're seeing, the strange thing is that a "List Resources" works fine so there is SNMP connectivity to the node.
I've popped Wireshark on the polling server and I see SNMP going out but nothing coming back apart from a TTL exceeded from our core site firewall (Cisco ASA5510).
The problem node continues to respond via ICMP whilst this is going on (it's connected via an ASA to ASA VPN tunnel).
I've restarted the firewall at the problem site but it's not made any difference and unfortunately I can't restart the core site firewall.
The thing that's throwing me is that if I restart the Orion services onthe polling server (or restart the server itself) the system start's polling the node correctly via SNMP again, making me think that it's something on the polling server that's going wrong.
Am I best opening a support ticket about this?
You've said that this was happening for two sites, each behind a different firewall. But if all the traffic is going through that core site firewall, it looks like the problem might be in there. That is really unfortunate that you can't restart it.
When this happens again, please try whether restarting only the Job Engine v2 service makes it start working again (so you don't have to restart everything).
And yes, please do open support ticket. We'll need to check the firewall configuration, hopefully there's something we can do to fix this.
Thanks Jan, I've just opened a support ticket.
I've restarted the Job Engine v2 service and the node is now being polled correctly, will wait to see what tech support say.