0 Replies Latest reply on Jun 4, 2015 9:06 AM by Anthony Ussery

    Process monitor over-monitoring

    Anthony Ussery

      Good morning,

       

      I've got a Linux process monitor monitoring the snmpd process, and an alert set to go off if the process goes Unknown.

      Unfortunately, in most cases where the service goes Unknown, the SNMP agent is still running but refusing to answer connection because it has too many files open and more requests coming in literally every second.  Why such aggressive follow-up, and can I dial that back to give the server time to process?

       

      Here's an example, separated out by second:

       

      Jun  4 06:41:34 HOST snmpd[7624]: warning: cannot open /etc/hosts.allow: Too many open files

      Jun  4 06:41:34 HOST snmpd[7624]: warning: cannot open /etc/hosts.deny: Too many open files

      Jun  4 06:41:34 HOST snmpd[7624]: Connection from UDP: [64.233.160.106]:56374 REFUSED

      Jun  4 06:41:34 HOST snmpd[7624]: /proc/stat: Too many open files

      Jun  4 06:41:34 HOST snmpd[7624]: could not open /proc/net/arp

      Jun  4 06:41:34 HOST snmpd[7624]: Unable to create netlink socket

       

      Jun  4 06:41:35 HOST snmpd[7624]: warning: cannot open /etc/hosts.allow: Too many open files

      Jun  4 06:41:35 HOST snmpd[7624]: warning: cannot open /etc/hosts.deny: Too many open files

      Jun  4 06:41:35 HOST snmpd[7624]: Connection from UDP: [64.233.160.106]:61451 REFUSED

       

      Jun  4 06:41:36 HOST snmpd[7624]: warning: cannot open /etc/hosts.allow: Too many open files

      Jun  4 06:41:36 HOST snmpd[7624]: warning: cannot open /etc/hosts.deny: Too many open files

      Jun  4 06:41:36 HOST snmpd[7624]: Connection from UDP: [64.233.160.106]:64073 REFUSED

       

      Jun  4 06:41:37 HOST snmpd[7624]: warning: cannot open /etc/hosts.allow: Too many open files

      Jun  4 06:41:37 HOST snmpd[7624]: warning: cannot open /etc/hosts.deny: Too many open files

      Jun  4 06:41:37 HOST snmpd[7624]: Connection from UDP: [64.233.160.106]:61451 REFUSED

       

      Jun  4 06:41:39 HOST snmpd[7624]: /proc/stat: Too many open files

       

      Jun  4 06:41:40 HOST snmpd[7624]: warning: cannot open /etc/hosts.allow: Too many open files

      Jun  4 06:41:40 HOST snmpd[7624]: warning: cannot open /etc/hosts.deny: Too many open files

      Jun  4 06:41:40 HOST snmpd[7624]: Connection from UDP: [64.233.160.106]:64073 REFUSED

       

      Jun  4 06:41:41 HOST snmpd[7624]: warning: cannot open /etc/hosts.allow: Too many open files

      Jun  4 06:41:41 HOST snmpd[7624]: warning: cannot open /etc/hosts.deny: Too many open files

      Jun  4 06:41:41 HOST snmpd[7624]: Connection from UDP: [64.233.160.106]:56711 REFUSED

       

      Jun  4 06:41:44 HOST snmpd[7624]: /proc/stat: Too many open files

       

      Jun  4 06:41:46 HOST snmpd[7624]: warning: cannot open /etc/hosts.allow: Too many open files

      Jun  4 06:41:46 HOST snmpd[7624]: warning: cannot open /etc/hosts.deny: Too many open files

      Jun  4 06:41:46 HOST snmpd[7624]: Connection from UDP: [64.233.160.106]:64073 REFUSED

       

      Jun  4 06:41:47 HOST snmpd[7624]: warning: cannot open /etc/hosts.allow: Too many open files

      Jun  4 06:41:47 HOST snmpd[7624]: warning: cannot open /etc/hosts.deny: Too many open files

      Jun  4 06:41:47 HOST snmpd[7624]: Connection from UDP: [64.233.160.106]:56711 REFUSED

       

      Jun  4 06:41:48 HOST snmpd[7624]: warning: cannot open /etc/hosts.allow: Too many open files

      Jun  4 06:41:48 HOST snmpd[7624]: warning: cannot open /etc/hosts.deny: Too many open files

      Jun  4 06:41:48 HOST snmpd[7624]: Connection from UDP: [64.233.160.106]:64073 REFUSED

       

      Jun  4 06:41:49 HOST snmpd[7624]: /proc/stat: Too many open files

      Jun  4 06:41:49 HOST snmpd[7624]: warning: cannot open /etc/hosts.allow: Too many open files

      Jun  4 06:41:49 HOST snmpd[7624]: warning: cannot open /etc/hosts.deny: Too many open files

      Jun  4 06:41:49 HOST snmpd[7624]: Connection from UDP: [64.233.160.106]:56711 REFUSED

      Jun  4 06:41:49 HOST snmpd[7624]: could not open /proc/net/arp

      Jun  4 06:41:49 HOST snmpd[7624]: Unable to create netlink socket

       

      Jun  4 06:41:52 HOST snmpd[7624]: warning: cannot open /etc/hosts.allow: Too many open files

      Jun  4 06:41:52 HOST snmpd[7624]: warning: cannot open /etc/hosts.deny: Too many open files

      Jun  4 06:41:52 HOST snmpd[7624]: Connection from UDP: [64.233.160.106]:64073 REFUSED

       

      Jun  4 06:41:53 HOST snmpd[7624]: warning: cannot open /etc/hosts.allow: Too many open files

      Jun  4 06:41:53 HOST snmpd[7624]: warning: cannot open /etc/hosts.deny: Too many open files

      Jun  4 06:41:53 HOST snmpd[7624]: Connection from UDP: [64.233.160.106]:56711 REFUSED

       

      Jun  4 06:41:54 HOST snmpd[7624]: /proc/stat: Too many open files

       

      Jun  4 06:41:56 HOST snmpd[7624]: cannot open /proc/net/dev ...

      Jun  4 06:41:56 HOST snmpd[7624]: could not create socket

       

      Jun  4 06:41:58 HOST snmpd[7624]: warning: cannot open /etc/hosts.allow: Too many open files

      Jun  4 06:41:58 HOST snmpd[7624]: warning: cannot open /etc/hosts.deny: Too many open files

      Jun  4 06:41:58 HOST snmpd[7624]: Connection from UDP: [64.233.160.106]:64073 REFUSED

       

      Jun  4 06:41:59 HOST snmpd[7624]: warning: cannot open /etc/hosts.allow: Too many open files

      Jun  4 06:41:59 HOST snmpd[7624]: warning: cannot open /etc/hosts.deny: Too many open files

      Jun  4 06:41:59 HOST snmpd[7624]: Connection from UDP: [64.233.160.106]:56711 REFUSED

      Jun  4 06:41:59 HOST snmpd[7624]: /proc/stat: Too many open files

       

      Jun  4 06:42:00 HOST snmpd[7624]: warning: cannot open /etc/hosts.allow: Too many open files

      Jun  4 06:42:00 HOST snmpd[7624]: warning: cannot open /etc/hosts.deny: Too many open files

      Jun  4 06:42:00 HOST snmpd[7624]: Connection from UDP: [64.233.160.106]:64073 REFUSED

       

      Jun  4 06:42:01 HOST snmpd[7624]: warning: cannot open /etc/hosts.allow: Too many open files

      Jun  4 06:42:01 HOST snmpd[7624]: warning: cannot open /etc/hosts.deny: Too many open files

      Jun  4 06:42:01 HOST snmpd[7624]: Connection from UDP: [64.233.160.106]:56711 REFUSED

       

      Jun  4 06:42:04 HOST snmpd[7624]: warning: cannot open /etc/hosts.allow: Too many open files

      Jun  4 06:42:04 HOST snmpd[7624]: warning: cannot open /etc/hosts.deny: Too many open files

      Jun  4 06:42:04 HOST snmpd[7624]: Connection from UDP: [64.233.160.106]:64073 REFUSED

      Jun  4 06:42:04 HOST snmpd[7624]: /proc/stat: Too many open files

      Jun  4 06:42:04 HOST snmpd[7624]: could not open /proc/net/arp

      Jun  4 06:42:04 HOST snmpd[7624]: Unable to create netlink socket

       

      Jun  4 06:42:05 HOST snmpd[7624]: warning: cannot open /etc/hosts.allow: Too many open files

      Jun  4 06:42:05 HOST snmpd[7624]: warning: cannot open /etc/hosts.deny: Too many open files

      Jun  4 06:42:05 HOST snmpd[7624]: Connection from UDP: [64.233.160.106]:56711 REFUSED

       

      Jun  4 06:42:08 HOST snmpd[7624]: net-snmp: 8 error(s) in config file(s)

       

      Jun  4 06:42:09 HOST snmpd[7624]: /proc/stat: Too many open files

       

      Jun  4 06:42:10 HOST snmpd[7624]: warning: cannot open /etc/hosts.allow: Too many open files

      Jun  4 06:42:10 HOST snmpd[7624]: warning: cannot open /etc/hosts.deny: Too many open files

      Jun  4 06:42:10 HOST snmpd[7624]: Connection from UDP: [64.233.160.106]:55797 REFUSED

       

      Jun  4 06:42:12 HOST snmpd[7624]: warning: cannot open /etc/hosts.allow: Too many open files

      Jun  4 06:42:12 HOST snmpd[7624]: warning: cannot open /etc/hosts.deny: Too many open files

      Jun  4 06:42:12 HOST snmpd[7624]: Connection from UDP: [64.233.160.106]:55797 REFUSED

       

      Jun  4 06:42:13 HOST snmpd[7624]: warning: cannot open /etc/hosts.allow: Too many open files

      Jun  4 06:42:13 HOST snmpd[7624]: warning: cannot open /etc/hosts.deny: Too many open files

      Jun  4 06:42:13 HOST snmpd[7624]: Connection from UDP: [64.233.160.106]:52878 REFUSED

       

      Jun  4 06:42:14 HOST snmpd[7624]: /proc/stat: Too many open files

      Jun  4 06:42:14 HOST snmpd[7624]: read_config_store open failure on /var/net-snmp/snmpd.conf