1 Reply Latest reply on Oct 6, 2014 11:38 AM by curtisi

    LEM: Is there a way to delete old data from the alerts database and/or set the number of days retention such that older alerts are automatically deleted?

    tcrutchley

      I have close to two years worth of data in my LEM now.  I'm also experiencing slow response-times and/or timeouts when executing nDepth searches.  I'm guessing this is directly relating to the size of the alerts database.

        • Re: LEM: Is there a way to delete old data from the alerts database and/or set the number of days retention such that older alerts are automatically deleted?
          curtisi

          I see this question a lot, so here's some info.  There are two ways to approach retention in an appliance like the LEM.

           

          First: you define a number of days.  If the device can keep that number with the assigned resources, all is well.  If it can't, then it starts to scream it's head off about how it needs more disk space.  How does it handle new traffic if it can't reach the desired retention span?  Either it stops collecting new events, or it starts pushing stuff out while it waits for you to do maintenance.

           

          Second: you define a disk size.  The device keeps as many events as it can in the space you gave it.  If you need a longer retention span, you either reduce the traffic or expand the disk.  The device rotates the database at some percentage of the disk to drop the oldest events and keep newer events.

           

          The LEM works with method two.  Retention is determined (roughly) by [size of alertDB partitions/size of daily event traffic].  When it reaches 90% full, the LEM starts rotating the database.  If you want the retention span of a LEM, you need to run a Database Maintenance Report in the Reports Console, and look at the last page for the span (my lab doesn't rotate very fast with so little traffic):

           

          2014-10-06 10_34_02-SolarWinds Log & Event Manager Reports.png

          Now, slowness: I don't think the database size and the performance are linked, so:

          • Are you running searches for very large time ranges?
          • Do these searches involve millions of records or results (ie, user logons/logoffs/failures for 90 days)?
          • Are you searching in the distant past?

           

          The LEM highly compresses partitions that are older, which is part of how the retention spans are achieved on small disks.  By default, the LEM only keeps about a week of data "warm" or decompressed for fast searching.  If you're searching more than a week in the past, the LEM has to pull archives into memory, decompress them, run your query, and then recompress them.  This can slow down searches.  Try searching a smaller time-frame (ie, a week vs a month) or refining the search to pull fewer results.

           

          It may also be simple resourcing: what memory/CPUs do you have assigned to the LEM?  What reservations do you have set?