3 Replies Latest reply on Dec 17, 2010 2:18 PM by daricsmith

    Problem with Alerts and db disparity

    daricsmith

      So I'm trying to figure out if I just have a screwed up Orion 10 SP1 installation (and causing goofiness with my Orion) or I'm doing something wrong.  I'm hoping it's the latter.

       

      I am trying to set an alert that will notify me when the percent used HD space changes on a server.  So I set up the alert for "Volume" equal to "Fixed Disk".  Then I add another simple condition saying to trigger if the "Volume Percent Available" ->  "Has Changed".  That is it.  A very simply query, just the 2 conditions.

      The alert is set up to activate in NPM as well as e-mail me a notice of the volume change.  If I test the alert, it works: The event manager shows the trigger and I get an e-mail notification.

      My next test is copying a very large amount of data from the network to the hard drive.  Basically, the HD sits at 9% used and over a period of 30 minutes, it will jump up to 22% used (copying about 40G of data).

      Problem is: Nothing triggers.  I monitor the volume.  I have the polling interval set to 120 seconds, and also set my web console to refresh every 2 minutes.  Nothing.  The percent used goes up, but I don't see any event triggered.

      Now here is the other goofy thing: If I open the Solarwinds Database Manager, select the "Alerts" Table and run a SQL query to show me all of my alerts: The list in the DB doesn't even match the list of alerts I have configured (see screen shots).  Is there another table alerts are stored in or something or is my installation screwed up?

        • Re: Problem with Alerts and db disparity
          smargh

          This is the TriggerQuery from the AlertDefinitions table for your alert. Notice the last bit - "last changed" is only for use with LastBoot.

           

          SELECT DISTINCT Volumes.VolumeID AS NetObjectID,
                          Volumes.FullName AS Name
          FROM Volumes
          WHERE ((Volumes.VolumeType = 'Fixed Disk') AND (100-NullIf(VolumePercentUsed,-2) = 100-NullIf(VolumePercentUsed,-2)

          and ((select count(*) from AlertValueChanges where TableName = 'Nodes' and FieldName = 'LastBoot') > 0) and NodeID in (select NodeID from AlertValueChanges where TableName = 'Nodes' and FieldName = 'LastBoot' and (DateDiff(s, Changed, getdate()) < (60 * 1.2)))))  )   

          • Re: Problem with Alerts and db disparity
            daricsmith

            So, even though I can set up a conditional with "Has Changed" (Not "Last Changed" as you stated in your post) as an option for "Percent Volume Available" That particular query doesn't actually work?

            Is there any way for me to get a trigger to occur when the volume amount changes?  Ideally I would like to set it up so that if the volume changes over a 5+ minute time-frame, I get an alert.

            The reason I am looking to do this is we had an AS400 server that went from 50% volume utilization to 100% over a 6 hour period filling up with error logs.  Yes, having a notification when the HD space is low helps, but having a notification when the volume changes dramatically can also help.

             

            Thank you

             

            -Daric

            • Re: Problem with Alerts and db disparity
              daricsmith

              OK, I fixed my problem by creating my own SQL solution. 

               

              As was pointed out: Orion Alerts are unable to trigger based "volume -> has changed".  I'm no SQL whiz so I am sure you could generate a query from the Volume_Details table, but I went for an easy fix.

              In the "Volumes" table, I created 2 new columns: VolumeLastPolledPercent AND VolumeChangeType.

              The "VolumeLastPolledPercent" would simply contain a copy of the column "VolumePercentUsed" that I have a script updating every 30 minutes.

              The "VolumeChangeType" column has three values: Increase, Decrease or Same.  I have a script that runs every 2 minutes to compare the column values in "VolumePercentUsed" and "VolumeLastPolledPercent".  I currently have the script only act if the difference is greater than +-2 percent.  If it is within the range of -2 to +2, the VolumeChangeType is set to "Same" as in the volumepercent hasn't changed significantly.

              But, let's say a large amount of data suddenly gets added to a volume so the percent used goes from 56% up to 60% over a 5 minute period.  I now have my alert using a SQL query that checks on the data in "VolumeChangeType".  If the change type is set to "Increase" the alert fires and notifies me that the HD space is quickly running out of space.

              But, if that amount of data levels off at 60%, the script that copies the values from one column to another runs at the 30 minute interval then the script that compares the differences of the 2 columns runs and resets the "VolumeChangeType" back to same.  The trigger is then reset.

              The "Decrease" value is for tracking when large amounts of data are deleted.  I can be alerted then when that happens.

               

              Again, the reason for all of this is to monitor a server in case the volume suddenly runs out of space (as happened with us).  Before the volume is at 100% and crashes the server, we can hopefully fix the problem.

               

              Anyway, I know it's a mouthful but I hope people find it useful.  If anyone would like a copy of the SQL queries along with the VB scripts that I run, I'd be happy to post those up.