This discussion has been locked. The information referenced herein may be inaccurate due to age, software updates, or external references.
You can no longer post new replies to this discussion. If you have a similar question you can start a new discussion in this forum.

Alerting on Volume Growth


Hello! I am new to using Solarwinds but have manged to create some basic alerts. One thing I would like to be able to do is alert if volume growth spikes above a certain level. I know I can alert if a volume exceeds a percentage or fixed amout of space etc but I think real value would come from knowing if a volume suddenly starts increasing at a rapid rate. Anyone know if this is possible?

Thanks!

  • What would you consider a rapid rate?

  • I would also like to know how to alert for rapid increase in disk usage.

    I would consider rapid growth to be an increase of 10% disk usage over the past 30 minutes.

    Along with alerting for the rapid disk usage growth, can I specify what or who is causing the issue.

  • Was there ever any alert created for this?

  • I have this same question. Did anyone find a solution for this?

  • It has been done in custom SQL/SWQL queries.  Basically you need to determine the size of your window (lets say an hour), what growth percent would you consider "alarming" (lets say 10% over that hour)  and then write a query to find the value from an hour ago, divide that by the current value, and if the result is greater than 1.1 trigger an alert. 

    In practice though this kind of alerts generate WAY more heat than light where I have seen them in use. 

    For example, I would trigger them all the time while doing solarwinds upgrades since the installers tend to be a one time jump of several GB on disk.  If you make the time window too large you miss the things you are worried about like db logs suddenly filling up, but if you make the window too narrow you have problems with one time bumps.  In order to make this type of alert more useful I have had to do things like check 3-4 timestamps along the window and try to measure if the growth trend stays high across all of them (also keeping in mind that disk space is only polled once every 15 minutes in most environments).  All told it gets to be pretty complex, but doable.

    The SAM Appinisght for SQL template does daily growth rate calculations for you automatically so you could use that for your DB or logs too if that is the use case you have in mind, but it is doing it across 24 hours so it has limited utility in catching those fast bursts.

  • Hi

    Try to make an alert out of this SQL query. (Createa a Custom SQL alert and choose Volumes as "target"):

    SELECT Volumes.FullName, Volumes.VolumeID FROM Volumes

    Inner join [dbo].[VolumeUsage_Detail]  D on Volumes.VolumeID=d.VolumeID

    where

    D.datetime>Dateadd(HOUR,-2,Getdate())  -- In how many hours

    AND Volumes.VolumeTypeID=4

    group by Volumes.FullName, Volumes.VolumeID

    HAVING Max(D.PercentDiskUsed)-Min(D.PercentDiskUsed)>10  -- Growth to look for in percent

    Adjust hours and percent growth as you want.

    Good luck!

  • Thanks for the great information mesverrum​ & Seashore​!!

  • A Bit of a variation of the script above - where the script looks at the last two polls to alert on a spike in % used. Also has a reset query to set a threshold for the reset.

      Volume Usage Spike Detection Alert

  • Here is a swql version. But it looks like it triggers on any difference, up or down that meets the threshold. I am still sorting that bit out.

    SELECT v.FullName, v.VolumeID, AVG(vh.PercentDiskUsed) as DISK_Used

    FROM Orion.Volumes v

    JOIN Orion.VolumeUsageHistory vh

        on v.VolumeID = vh.VolumeID

    WHERE vh.datetime> ADDHOUR(-2, GETUTCDATE())

    AND v.VolumeTypeID=4

    group by v.FullName, v.VolumeID

    HAVING Max(vh.PercentDiskUsed)-Min(vh.PercentDiskUsed)>10

  • First  all, thanks everyone for your posts which gave me the basis for what I was looking for which was alerting on Linux swap volume growth that gets out of control.

    Second, we needed more flexibility than a single hard-coded Percent Growth and Time Interval.  So by adding the following two Custom Properties to volumes, you can add the flexibility to either use the "defaults" or customize the parameters on a per volume basis:

        

         VolumeSizePercentageChangeTrigger          Floating Point       Percent Change to be used as a Trigger condition (default = 50.00)
         VolumeSizeChangeRateIntervalInMinutes     Int                         Interval (in minutes) to "look back" in time for the trigger percent trigger condition (default = 30 days)
    Here is the custom SQL query to achieve this:
    SELECT Volumes.[VolumeID], Volumes.[FullName] FROM [Orion.Solarwinds].[dbo].[Volumes]
        WHERE Volumes.VolumeTypeID IN ('4','10','100')                                                                                                                            -- Fixed Disk; Network Disk; Mount Point
            AND (SELECT CASE WHEN (SELECT (SELECT MAX(D.PercentDiskUsed) FROM [dbo].[VolumeUsage_Detail] AS D WHERE D.VolumeID = Volumes.[VolumeID])                            -- Get Max PercentDiskUsed
                            - (SELECT MIN(D.PercentDiskUsed) FROM [dbo].[VolumeUsage_Detail]  AS D WHERE D.VolumeID = Volumes.[VolumeID] AND D.datetime >                         -- Minus Min PercentDiskUsed
                                Dateadd(MINUTE,(-1 * (Select CASE WHEN ((SELECT C.VolumeSizeChangeRateIntervalInMinutes FROM [dbo].[Volumes] C                                    -- Get VolumeSizeChangeRateIntervalInMinutes
                                        WHERE C.VolumeID = Volumes.[VolumeID]) IS NULL)
                                    THEN '43200' ELSE (SELECT C.VolumeSizeChangeRateIntervalInMinutes FROM [dbo].[Volumes] C WHERE C.VolumeID = Volumes.[VolumeID])                -- EDIT DEFAULT INTERVAL IN MINUTES (30 Days) HERE
                                        END)),Getdate())))               
                            > (SELECT CASE WHEN ((SELECT C.VolumeSizePercentageChangeTrigger FROM [dbo].[Volumes] AS C WHERE C.VolumeID = Volumes.[VolumeID]) IS NULL)
                                THEN '50.00' ELSE (SELECT C.VolumeSizePercentageChangeTrigger FROM [dbo].[Volumes] AS C WHERE C.VolumeID = Volumes.[VolumeID]) END)                -- EDIT DEFAULT PERCENTAGE (50.00) CHANGE HERE
                        THEN (SELECT 'True') ELSE (SELECT 'False')
                    END AS 'Percent Growth Exceeds Set Point'   
            ) = 'True'

    VolumeSizeChangeRateIntervalInMinutes