4 Replies Latest reply on Mar 7, 2014 4:15 PM by byrona

    LEM Log Archiving?

    byrona

      I am curious if there is a way to archive your logs off LEM in such a way that it moves the logs out of the LEM database and into an archive freeing up the space in the LEM database?

       

      I am thinking about an architecture where you have LEM running on some high performance storage managing your logs but to maintain a specific retention policy having the ability to archive those logs out of LEM and save them on some slower less expensive storage.  This way you would have all of the benefits of LEM but also maintain any necessary (and possibly ridiculous) retention policy.

       

      Thanks in advance for any responses!

        • Re: LEM Log Archiving?
          nicole pauls

          Hmm, alas no, not exactly.

           

          The LEM archive of the normalized data will basically replicate any data that isn't already on the archive store, and it'll grow forever (more or less, as long as it can push that archive data off). But, it's a mirror/copy, not a move and delete sort of operation. If you used that model, you would leave your LEM appliance at one storage level, then use the archive for slower/long term/wasteful storage. If you had to pull something back in, it gets a little complicated, we'd need to figure out what to pull back in and probably do it on a secondary system to not affect/confuse the running system. (We're looking at ways to improve that bit, either making it easier to have a secondary "I just need to search these dates for a little while out of my backup" or use the remote archive directly as frozen storage).

            • Re: LEM Log Archiving?
              byrona

              Yeah, my thought was spinning up a 2nd appliance and importing the data into there if no other supported model was available.

               

              One idea going forward might be to have LEM connect to the archive file via a network share and just reading it over that versus importing it at all.  The idea is that these should only ever need to be accessed in special situations anyway and then detached shortly after so performance would not need to be the primary focus, just the ability to have it available.  Thoughts?

                • Re: LEM Log Archiving?
                  nicole pauls

                  That's what I was thinking as well with using the remote storage directly, I just don't know if the performance would be good enough over CIFS for it to be usable for search/reporting. (All of the LEM archive/backup stuff is CIFS-driven currently.) Right now it's all expected to be on disk, but since the files are self-contained it's much more technically possible than with a relational database. That also means, though, that we could pull over selective archives, which is kind of a middle ground - you still pull them back to the appliance, but you only pull the dates you need.

                    • Re: LEM Log Archiving?
                      byrona

                      So I guess one way of working my way around it is once the DB is full and throwing out the oldest events I can do a new backup every week.  While that wouldn't empty out the active LEM DB it would provide the longer term retention that could then be stored on slower disks... unless I am missing something obvious?