8 Replies Latest reply on Mar 14, 2018 2:49 PM by jdwinns

    Log Parser (Powershell)

    jdwinns

      Hello everyone,

       

      I currently have the Powershell Log parser up and running and collecting metrics on multiple log files spanning across multiple nodes in our environment. It's been working excellent thus far but I've noticed a couple small hiccups in the generated alerts.

       

      Normally when the alert fires, it will display the specific lines from the log file that caused the alert to trigger. This is done by using the string below in the alert email action which pulls the values from dbo.APM_AlertsAndReportsData. Also, side note, huge shout out to njoylif for helping me get this configured!! Thanks again man!

       

      ${SQL:SELECT CONCAT('','<table><tr><td>', REPLACE(REPLACE(MultiValueMessages , '; 20' ,';<br />20'), 'Lines that have search string:' ,'Lines that have search string:<br />'),'</td></tr></table>','') FROM SolarWindsOrion.dbo.APM_AlertsAndReportsData where ComponentID = ${ComponentID}}

       

      The issue I'm having is rare, maybe 1 out of 200 alerts. Occasionally, a log alert will fire and instead of getting the strings that caused the alert, we get "No newly found strings" as the output. I'm not sure if this is related to the temp file that the powershell script uses not getting refreshed properly? Or if the location where we are pulling the "MultiValueMessages" in the database isn't populated fast enough so it's just picking up "No newly found strings"?

       

      Anyhow, I'm hoping someone else had ran into this and could shed some light on it for us. Thank you for your time!

        • Re: Log Parser (Powershell)
          jdwinns

          I just noticed while troubleshooting that our database is out of sync. Not positive if the two issues are related or not. The "No newly found strings" alert issue has been happening on occasion for a few months, the database sync seems to be a newer problem.

           

          Has anyone else had this happen or know the reason why the database would be out of sync? Any possible resolutions? Server restart perhaps?

           

          db_sync

          • Re: Log Parser (Powershell)
            jdwinns

            Just an update on this - We continue to occasionally receive "No newly found strings" in alert emails from our "Log Parser - Newly Found Strings" applications. It happened today and I quickly verified that the alert was valid by locating the string in today's log.

             

            What I still believe might be happening is when the monitor detects and error string, it trips the alert and fires off an email before the DB field “MultiValueMessages” gets populated, so it’s grabbing the default “No newly found strings” instead.

             

            I was going to try and add a short delay before the E-mail trigger action to see if that resolves the problem. Has anyone else seen this happen? Is there a way to find out how much of a delay there is between when the component monitor parses the log and detects an ERROR string to how fast the database field “MultiValueMessages” gets populated?