10 Replies Latest reply on Mar 31, 2016 4:34 PM by bajcsi

    LEM / RHEL7 - No Log Data

    bajcsi

      Hello,

       

      Background

      We are just rolling out LEM (6.2) and have hit a speed bump while configuring our Linux infrastructure for LEM.  After installing the linux agent on a RHEL 7 box (first one we've tried, and our current standard), I configured the connectors for that node via the LEM console. I was also mindful to reference appropriate log locations. Most are set to use alerts and nDepth. At least one is just set to alerts.

       

      However, unlike our syslog hosts and Windows agents, I have yet to see ANY log data with the exception of the agent restart process.  On my firewall, I can verify I see the the connection from host to lem (port 37892) is torn down when I stop the agent and rebuilt when I restart it.  I tested with both security events and installing a package via Yum.

       

      Please see the node configuration in this screenshot...

       

      LEM Console - Linux Agent.png

       

      Note only Agent start and Agent stop events are passed.  Running user is root.

       

      Digging Deeper

      lsof shows that the java process 24734 has /var/log/yum.log open:

       

      java      24734 24862    root  154r      REG              253,0       101  134296534 /var/log/yum.log

       

      Using ps, verified that pid 24734 is indeed the agent:

       

      root     24734  1.8  4.0 1740876 158000 pts/0  Sl   16:26   0:08 /usr/local/contego/ContegoSPOP/../ContegoSPOP/jre1.7.0_80/bin/java -Djava.library.path=6.2.0\\lib -Dlogback.configurationFile=jar:file:6.2.0/jars/lem_agent.jar!/logback-agent.xml com.zerog.lax.LAX /usr/local/contego/ContegoSPOP/SWLEMAgent.lax /tmp/env.properties.24734 "-lf" "/usr/local/contego/ContegoSPOP/agent.log"

       

      This host is not running SELinux, which is configured to be "disabled" and has been verified with getenforce.

       

      On my firewall (ASA running 9.4.x, I've allowed TCP/37890 (Install) and TCP/37892 (Secure connection).  With my log level set to Debug, I do not see any errors/blocked connections.

       

      Testing

      As mentioned above, I tested with both security events and installing a package via Yum. I've also configured a filter that shows me an resources from both the ip address and the hostname of the node.  The results are the same.

       

      I'd appreciate any guidance that could be offered.  I'm sure I missed something simple, but it's not at all obvious to me... Thanks in advance...

       

      Brett

        • Re: LEM / RHEL7 - No Log Data
          curtisi

          I've noticed that the Node Details widgets can be flaky.  If you do an nDepth search for AnyAlert.DetectionIP = THAT NODE, what do you get back for the last week?  Do you see YUM data there?

           

          We have Centos 7 running in the lab and it works with those connectors.  I know it's not RHEL, but the logs are similar.

            • Re: LEM / RHEL7 - No Log Data
              bajcsi

              Hi,

              So I searched through nDepth and was able to find some events for that IP address (which is weird, because I had last week as well). They still don't show through the Ops Center (per the above screenshot). Thoughts as to why they would show up under nDepth but not Ops Center?  The alerts are configured to go to both Alerts and nDepth as mentioned above. 

               

              As an aside, I have a RHEL 6 box on which I also installed the agent and I don't see it under the Ops Center or nDepth.  The connectors are, in theory, configured the same and correctly (pointing at the correct files, etc). 

               

              Thanks,

              Brett

                • Re: LEM / RHEL7 - No Log Data
                  curtisi

                  OpsCenter and Monitor are transient and real-time, so every time the console is reloaded, they all go back to zero and start counting again.  nDepth is how we search the permanent record the LEM creates, so historical data is always in the nDepth.

                    • Re: LEM / RHEL7 - No Log Data
                      bajcsi

                      Aye, I realize that.  I'm starting to agree with your comment on flaky widgets though. The majority of log entries I see from that host are audit events like login and logoff - I don't actually see the Yum log I would have expected. see below:

                       

                      Screen Shot 2016-03-23 at 10.29.57 AM.png

                       

                      As an aside, I have a RHEL 6 box on which I also installed the agent and I don't see it under the Ops Center or nDepth.  The connectors are, in theory, configured the same and correctly (pointing at the correct files, etc). I'll keep tinkering.  Any thoughts on that? I also see the files open:

                       

                      java      11152      root  168r      REG              253,0         54    3823626 /var/log/yum.log

                      java      11152      root  169r      REG              253,0    1720126    2015333 /var/log/audit/audit.log

                      java      11152      root  171r      REG              253,0    2859319    2021709 /var/log/maillog

                      java      11152      root  176r      REG              253,0  646052382    1839390 /var/log/slapd.log

                       

                       

                      Thanks,

                      Brett

                        • Re: LEM / RHEL7 - No Log Data
                          curtisi

                          Silly question: have you run an install or update with YUM on those systems? There are current logs since the LEM Agent started reading for YUM activity?

                            • Re: LEM / RHEL7 - No Log Data
                              bajcsi

                              Hey,

                              Regarding the RHEL 7 box. Yes, I executed an install/remove sequence on a package to cause logging to the Yum log.

                               

                              [A01 log]# cat yum.log

                              Mar 23 10:25:34 Erased: lsof-4.87-4.el7.x86_64

                              Mar 23 10:25:48 Installed: lsof-4.87-4.el7.x86_64

                              [root@A01 log]#

                               

                              With regards to the above referenced RHEL6 box, there is definitely usage on the box as it is a heavily utilized box.  Yum is just my example, and I did generate log messages by installing/removing a package, but as you can see from the above screenshot we are watching the audit log, slapd logs, yum and maillog.  SOMETHING should have been forwarded. I'll I see when looking in nDepth for logs from the last week are the two times the agent  was started.