Linux Monitorring via SNMP vs Agent + SNMP

I've encountered an issue, specifically on a handful of RHEL 7 boxes, where I can add a node and configure monitoring using SNMPv3 without an issue. After SNMP monitoring had been in place for a bit the agent was installed on these hosts and I've switched them to use the agent polling method. This works. However, if I attempt to load the same SNMPv3 saved credential set and use SNMP settings as the supplement to the agent polling I wind up with a generic error message that doesn't give me much to work with.

Would anyone shed some insight into why SNMP polling might work, but agent + SNMP polling fails with an error of Test Failed. Test execution failed

Find more posts tagged with

Accepted answers

All comments

hpstech

Do you have APEs? Can you do a MIB walk? Firewall rules blocking?

Success Center

wittyusername

I do have APE's. I just tested one of the problem hosts. I can pull info using SNMPWalk from either the MPE and the APE that is supposed to poll this item. Firewall rules are not blocking traffic.

As I mentioned in my first post, SNMP and ICMP monitoring works as expected. Agent monitoring works, but when I attempt to Include SNMP Credentials the error message is thrown.

mesverrum

Just a guess, but maybe the localhost address isn't being allowed to query snmp? Can you try to launch an snmpwalk from the server to itself? If thats not the problem then I'd be looking for log files relating to the agents, either on the polling engine or locally on the node itself.

wittyusername

I was able to track down an error in the Core.BusinessLayer.log file on the MPE. It seems like this is more of a JobEngine issue since the same SNMP credential set that doesn't work for Agent monitoring works fine for SNMP and ICMP monitorring.

2019-12-02 11:39:26,481 [115] WARN SolarWinds.Orion.Core.Common.JobEngine.JobScheduler - (null) AddJob: JobScheduler channel threw FaultException

2019-12-02 11:39:26,481 [115] WARN SolarWinds.Orion.Core.Common.JobEngine.JobScheduler - (null) Job Scheduler Exception:

System.ServiceModel.FaultException`1[SolarWinds.JobEngine.JobEngineConnectionFault]: Unable to communicate with agent JobEngine on endpoint e10c547f-0dc8-41j6-8b8a-62e5a320221e. Response to message 'AddJob' was not received in '00:02:00'. (Fault Detail is equal to SolarWinds.JobEngine.JobEngineConnectionFault).

2019-12-02 11:39:26,481 [115] ERROR SolarWinds.Orion.Core.BusinessLayer.OneTimeJobs.OneTimeJobManager - (null) Failed to submit job: System.ServiceModel.FaultException`1[SolarWinds.JobEngine.JobEngineConnectionFault]: Unable to communicate with agent JobEngine on endpoint e10c547f-0dc8-41j6-8b8a-62e5a320221e. Response to message 'AddJob' was not received in '00:02:00'. (Fault Detail is equal to SolarWinds.JobEngine.JobEngineConnectionFault).

2019-12-02 11:39:26,481 [115] WARN SolarWinds.Orion.Core.BusinessLayer.CoreBusinessLayerService - (null) SNMP credential test failed: Test execution failed

pratikmehta003

any specific reason that u want to include SNMP even though agent is going to get all details?

apocalyps3

My understanding is that you can't pull Asset Inventory information from Linux systems that are agent only. It'll only pull Asset Inventory if SNMP (v3) is configured.

I've tested this with a couple systems and it seems to be true.

Those systems with just the Agent: no Asset Inventory

Those systems with SNMPv3: Asset Inventory

tomasv

Did you try to configure SNMP daemon on the target machine for SNMP v2 - with any community string - with no remote access to the SNMP ports?

tomasv

Asset Inventory collects the details via SNMP even it runs using agent. So the approach is the same for agent-less and agent polling. So it gets the same asset data.

The difference is, that SNMP requests run locally from Solarwinds agent - so it needs no remote access to the SNMP ports.

apocalyps3

tomasv wrote:
Asset Inventory collects the details via SNMP even it runs using agent. So the approach is the same for agent-less and agent polling. So it gets the same asset data.
The difference is, that SNMP requests run locally from Solarwinds agent - so it needs no remote access to the SNMP ports.

I've read this elsewhere on the forum, but in practice I've never actually seen it work like this for our Linux systems.

I haven't had a single Linux system, with the agent installed, report back Asset Inventory data. I've never been able to get the Asset Inventory checkbox to appear under "List Resources" unless I've installed/configured SNMPv3 on the Linux system and set the Polling method to SNMP.

If the polling method is set to Agent, then Asset Inventory is never even an option. I don't know if I'm missing something simple, but I've done multiple testing of these specific scenarios and its always the case.

tomasv

Coul'd you share, without exposing sensitive data, how does /etc/snmp/snmpd.conf look like?

When you do list resources on your node, and then check the directory /opt/SolarWinds/Agent/bin/appdata/Logs

is there any error in file AssetInventory_job_pollers.log or Core_job_discovery.log ?

ahbrook

It feels like I spent literally all afternoon working on this, when I probably didn't need to.. But hey, best to pass my information onto others. Especially as Observability is coming, and agents will become more common.

Baseline: Environment info.
Solarwinds Orion + SAM, 2020.4
OS: Red Hat Enterprise Linux 8.1
Agent: 2.4.4.2285

Support websites / help pages referenced:
https://access.redhat.com/solutions/17815 - setting up snmpv3 on RHEL
https://support.solarwinds.com/SuccessCenter/s/article/Run-SolarWinds-SNMPWALK?language=en_US - Running SNMPWalk from Orion polling engines
https://documentation.solarwinds.com/en/success_center/sam/content/sam-troubleshooting-snmp.htm - setting up SNMP with Orion Agents (note it specifically talks about SNMPv3)
https://access.redhat.com/discussions/2607871 - installing lsb-core on a Red Hat box (see below for why)
https://support.solarwinds.com/SuccessCenter/s/article/Linux-Agent-cannot-enable-SNMP-polling?language=en_US - SNMPv3 Credentials Fail with Linux Agent
https://thwack.solarwinds.com/product-forums/server-application-monitor-sam/f/forum/35634/net-snmp-snmpv2-v3-configuration-example-for-linux-distros - Example SNMPv2/v3 configurations (but not for agents)
https://support.solarwinds.com/SuccessCenter/s/article/Troubleshoot-an-SNMP-Node-SNMP-Test-Failed-result?language=en_US - Troubleshooting SNMP failures

With all the links out of the way...

Long story short - I was in a scenario where snmpwalk was working for v3 from both Orion polling engines we have, and from localhost. I did have to open up firewall ports for udp 161, but it worked. The snmpwalk tests from the polling engines said my permissions were not high enough to scan some things, though.

Despite all these working, my "Test" button for the agent running snmpv3 did not work. I reconfigured with both DES and AES encryption, and nothing happened.

Looking through the logs as indicated in this thread, I found references to "lsb not found" (sadly I can't find those errors now for an example). This led me to discover that lsb-core is not installed by default on Red Hat boxes, but Orion was looking for it. I installed redhat-lsb via yum, and the error disappeared. But I still couldn't get the agent to poll.

At this point I tested SNMP with ICMP, and found everything worked. So the problem was with the agent config, not my SNMP config.

I kept digging through different settings to no avail, until I finally came across the "SNMPv3 Credentials Fail with Linux Agent" support article. This indicated that PyCryptodomex was not present in my install. I tried installing this (yum python3-PyCryptodomex) but the agent still failed the test validation button.

At this point, I fell back to setting up an SNMPv2c community string, limited to only the localhost. This was done using the example post from the community above.

Running Test "STILL" failed. However, I noticed that my node had the "Location" and "contact" details. Both of these are set via the snmpd.conf file... In other words, the agent was successfully reading the config and getting data. So I tried to add asset inventory information. SUCCESS!

So it looks like however the agent tests/validates will fail, but the system will still be able to pull in quite a bit of information. If you have things set up properly, that is.

Here's everything I did. Again, this is RHEL 8.

sudo yum install -y net-snmp net-snmp-utils net-snmp-libs net-snmp-devel
# These steps may not be required or desired
sudo yum install -y redhat-lsb python3-pycryptodomex

# Enabling snmpd at startup
sudo chkconfig snmpd on
sudo systemctl enable snmpd
sudo service snmpd restart

# Editing the snmpd.conf file (simplified to show examples, see example conf for what this all means)

sudo vim /etc/snmp/snmpd.conf
-----

com2sec OrionUser  default   [redacted community name]
group   OrionGroup v2c       Orionuser
view    OrionView  included  .1
access  OrionGroup "" any noauth exact OrionView none none

syslocation [our VM Datacenter]
syscontact [The email we use for alerts]

agentAddress udp:127.0.0.1:161
-----

# The agent needs to be restarted when the conf is changed.
sudo service snmpd restart

# Testing local snmp works
snmpwalk -v 2c -c [redacted community name] localhost .1

Ideally, this should "Just Work." The fact that a 2020 article specifically mentions that they know a python module isn't being installed, and as a result the agent isn't able to use encrypted v3 credentials, is just... weird. Further, the "test unsuccessful" error on the Node Configuration page is less than helpful, and for the life of me I couldn't find agent logs that pointed at the source of the error.

But hey. This is enough for me to get our server setup adjusted to automate setting up this local community, and extend the featureset of the agent. So hopefully this can help people.