Good morning,
I've got a Linux process monitor monitoring the snmpd process, and an alert set to go off if the process goes Unknown.
Unfortunately, in most cases where the service goes Unknown, the SNMP agent is still running but refusing to answer connection because it has too many files open and more requests coming in literally every second. Why such aggressive follow-up, and can I dial that back to give the server time to process?
Here's an example, separated out by second:
Jun 4 06:41:34 HOST snmpd[7624]: warning: cannot open /etc/hosts.allow: Too many open files
Jun 4 06:41:34 HOST snmpd[7624]: warning: cannot open /etc/hosts.deny: Too many open files
Jun 4 06:41:34 HOST snmpd[7624]: Connection from UDP: [64.233.160.106]:56374 REFUSED
Jun 4 06:41:34 HOST snmpd[7624]: /proc/stat: Too many open files
Jun 4 06:41:34 HOST snmpd[7624]: could not open /proc/net/arp
Jun 4 06:41:34 HOST snmpd[7624]: Unable to create netlink socket
Jun 4 06:41:35 HOST snmpd[7624]: warning: cannot open /etc/hosts.allow: Too many open files
Jun 4 06:41:35 HOST snmpd[7624]: warning: cannot open /etc/hosts.deny: Too many open files
Jun 4 06:41:35 HOST snmpd[7624]: Connection from UDP: [64.233.160.106]:61451 REFUSED
Jun 4 06:41:36 HOST snmpd[7624]: warning: cannot open /etc/hosts.allow: Too many open files
Jun 4 06:41:36 HOST snmpd[7624]: warning: cannot open /etc/hosts.deny: Too many open files
Jun 4 06:41:36 HOST snmpd[7624]: Connection from UDP: [64.233.160.106]:64073 REFUSED
Jun 4 06:41:37 HOST snmpd[7624]: warning: cannot open /etc/hosts.allow: Too many open files
Jun 4 06:41:37 HOST snmpd[7624]: warning: cannot open /etc/hosts.deny: Too many open files
Jun 4 06:41:37 HOST snmpd[7624]: Connection from UDP: [64.233.160.106]:61451 REFUSED
Jun 4 06:41:39 HOST snmpd[7624]: /proc/stat: Too many open files
Jun 4 06:41:40 HOST snmpd[7624]: warning: cannot open /etc/hosts.allow: Too many open files
Jun 4 06:41:40 HOST snmpd[7624]: warning: cannot open /etc/hosts.deny: Too many open files
Jun 4 06:41:40 HOST snmpd[7624]: Connection from UDP: [64.233.160.106]:64073 REFUSED
Jun 4 06:41:41 HOST snmpd[7624]: warning: cannot open /etc/hosts.allow: Too many open files
Jun 4 06:41:41 HOST snmpd[7624]: warning: cannot open /etc/hosts.deny: Too many open files
Jun 4 06:41:41 HOST snmpd[7624]: Connection from UDP: [64.233.160.106]:56711 REFUSED
Jun 4 06:41:44 HOST snmpd[7624]: /proc/stat: Too many open files
Jun 4 06:41:46 HOST snmpd[7624]: warning: cannot open /etc/hosts.allow: Too many open files
Jun 4 06:41:46 HOST snmpd[7624]: warning: cannot open /etc/hosts.deny: Too many open files
Jun 4 06:41:46 HOST snmpd[7624]: Connection from UDP: [64.233.160.106]:64073 REFUSED
Jun 4 06:41:47 HOST snmpd[7624]: warning: cannot open /etc/hosts.allow: Too many open files
Jun 4 06:41:47 HOST snmpd[7624]: warning: cannot open /etc/hosts.deny: Too many open files
Jun 4 06:41:47 HOST snmpd[7624]: Connection from UDP: [64.233.160.106]:56711 REFUSED
Jun 4 06:41:48 HOST snmpd[7624]: warning: cannot open /etc/hosts.allow: Too many open files
Jun 4 06:41:48 HOST snmpd[7624]: warning: cannot open /etc/hosts.deny: Too many open files
Jun 4 06:41:48 HOST snmpd[7624]: Connection from UDP: [64.233.160.106]:64073 REFUSED
Jun 4 06:41:49 HOST snmpd[7624]: /proc/stat: Too many open files
Jun 4 06:41:49 HOST snmpd[7624]: warning: cannot open /etc/hosts.allow: Too many open files
Jun 4 06:41:49 HOST snmpd[7624]: warning: cannot open /etc/hosts.deny: Too many open files
Jun 4 06:41:49 HOST snmpd[7624]: Connection from UDP: [64.233.160.106]:56711 REFUSED
Jun 4 06:41:49 HOST snmpd[7624]: could not open /proc/net/arp
Jun 4 06:41:49 HOST snmpd[7624]: Unable to create netlink socket
Jun 4 06:41:52 HOST snmpd[7624]: warning: cannot open /etc/hosts.allow: Too many open files
Jun 4 06:41:52 HOST snmpd[7624]: warning: cannot open /etc/hosts.deny: Too many open files
Jun 4 06:41:52 HOST snmpd[7624]: Connection from UDP: [64.233.160.106]:64073 REFUSED
Jun 4 06:41:53 HOST snmpd[7624]: warning: cannot open /etc/hosts.allow: Too many open files
Jun 4 06:41:53 HOST snmpd[7624]: warning: cannot open /etc/hosts.deny: Too many open files
Jun 4 06:41:53 HOST snmpd[7624]: Connection from UDP: [64.233.160.106]:56711 REFUSED
Jun 4 06:41:54 HOST snmpd[7624]: /proc/stat: Too many open files
Jun 4 06:41:56 HOST snmpd[7624]: cannot open /proc/net/dev ...
Jun 4 06:41:56 HOST snmpd[7624]: could not create socket
Jun 4 06:41:58 HOST snmpd[7624]: warning: cannot open /etc/hosts.allow: Too many open files
Jun 4 06:41:58 HOST snmpd[7624]: warning: cannot open /etc/hosts.deny: Too many open files
Jun 4 06:41:58 HOST snmpd[7624]: Connection from UDP: [64.233.160.106]:64073 REFUSED
Jun 4 06:41:59 HOST snmpd[7624]: warning: cannot open /etc/hosts.allow: Too many open files
Jun 4 06:41:59 HOST snmpd[7624]: warning: cannot open /etc/hosts.deny: Too many open files
Jun 4 06:41:59 HOST snmpd[7624]: Connection from UDP: [64.233.160.106]:56711 REFUSED
Jun 4 06:41:59 HOST snmpd[7624]: /proc/stat: Too many open files
Jun 4 06:42:00 HOST snmpd[7624]: warning: cannot open /etc/hosts.allow: Too many open files
Jun 4 06:42:00 HOST snmpd[7624]: warning: cannot open /etc/hosts.deny: Too many open files
Jun 4 06:42:00 HOST snmpd[7624]: Connection from UDP: [64.233.160.106]:64073 REFUSED
Jun 4 06:42:01 HOST snmpd[7624]: warning: cannot open /etc/hosts.allow: Too many open files
Jun 4 06:42:01 HOST snmpd[7624]: warning: cannot open /etc/hosts.deny: Too many open files
Jun 4 06:42:01 HOST snmpd[7624]: Connection from UDP: [64.233.160.106]:56711 REFUSED
Jun 4 06:42:04 HOST snmpd[7624]: warning: cannot open /etc/hosts.allow: Too many open files
Jun 4 06:42:04 HOST snmpd[7624]: warning: cannot open /etc/hosts.deny: Too many open files
Jun 4 06:42:04 HOST snmpd[7624]: Connection from UDP: [64.233.160.106]:64073 REFUSED
Jun 4 06:42:04 HOST snmpd[7624]: /proc/stat: Too many open files
Jun 4 06:42:04 HOST snmpd[7624]: could not open /proc/net/arp
Jun 4 06:42:04 HOST snmpd[7624]: Unable to create netlink socket
Jun 4 06:42:05 HOST snmpd[7624]: warning: cannot open /etc/hosts.allow: Too many open files
Jun 4 06:42:05 HOST snmpd[7624]: warning: cannot open /etc/hosts.deny: Too many open files
Jun 4 06:42:05 HOST snmpd[7624]: Connection from UDP: [64.233.160.106]:56711 REFUSED
Jun 4 06:42:08 HOST snmpd[7624]: net-snmp: 8 error(s) in config file(s)
Jun 4 06:42:09 HOST snmpd[7624]: /proc/stat: Too many open files
Jun 4 06:42:10 HOST snmpd[7624]: warning: cannot open /etc/hosts.allow: Too many open files
Jun 4 06:42:10 HOST snmpd[7624]: warning: cannot open /etc/hosts.deny: Too many open files
Jun 4 06:42:10 HOST snmpd[7624]: Connection from UDP: [64.233.160.106]:55797 REFUSED
Jun 4 06:42:12 HOST snmpd[7624]: warning: cannot open /etc/hosts.allow: Too many open files
Jun 4 06:42:12 HOST snmpd[7624]: warning: cannot open /etc/hosts.deny: Too many open files
Jun 4 06:42:12 HOST snmpd[7624]: Connection from UDP: [64.233.160.106]:55797 REFUSED
Jun 4 06:42:13 HOST snmpd[7624]: warning: cannot open /etc/hosts.allow: Too many open files
Jun 4 06:42:13 HOST snmpd[7624]: warning: cannot open /etc/hosts.deny: Too many open files
Jun 4 06:42:13 HOST snmpd[7624]: Connection from UDP: [64.233.160.106]:52878 REFUSED
Jun 4 06:42:14 HOST snmpd[7624]: /proc/stat: Too many open files
Jun 4 06:42:14 HOST snmpd[7624]: read_config_store open failure on /var/net-snmp/snmpd.conf
