3 Replies Latest reply on Dec 3, 2014 4:38 PM by tjreeddoc

    FEATURE REQUEST - Add options for archive compression


      I noticed recently that my monthly archiveconfigs have been taking about 20 hours from start to completion. After some time working with technical support (case# 330696), we determined the bottleneck seemed to be the with the gzip2 compression of the backup data on the appliance itself. In my case, I'm archiving about 260 GB of data which then gets compressed to about 130 GB, all of this ends up on a network file share. I appreciate the good compression ratio but would like to have the option to adjust the compression to save time and processing power.


      Some possible feature suggestions:

      • Allow compression options to be selected when configuring backup jobs to speed up the process by sacrificing disk space.
      • Add the ability to point the backup job to a LEM node by it's agent name, then add compression functionality to the agent installed on the backup computer of choice. This way the backup files can be copied to the machine of choice only once, then compression can be performed locally on the backup machine without the need to copy the data back and forth three times.
        • Re: FEATURE REQUEST - Add options for archive compression
          nicole pauls

          This is a great request. Archiving is something we want to spend some time on.


          We switched our nDepth log storage to archive two ways, which might be useful to implement for the database instead of the current "make me a big ol' copy" archive:

          1. Via a mechanism that copies data partitions/shards to a file share as they are finalized on the LEM appliance. As a data partition/shard is completed (i.e. will never be written to again), it is copied to a network share. The data's already compressed, so it doesn't get compressed again, it just gets copied over. It's sort of like an incremental backup, but it's not on a schedule, it happens as the data is written to disk. In high volume environments that could be multiple times a day, but for most people, it'd be days. We don't ever delete things from the network share, because we assume you want this to live forever (as a permanent-ish or longer term archive), and you will maintain the storage.
            1. NOTE: In the future, we've talked about being able to access these older data partitions/shards as a sort of remote "deep storage" search.
          2. Via a mechanism that synchronizes the current database directory to a file share on a daily basis. If data partitions/shards were deleted from the LEM Appliance for space reasons, we'll remove them from the file share also, so it's a constant living copy of what's on the appliance. This is more of a disaster recovery sort of thing, but it will only copy over what's new, sort of like a differential or incremental backup.


          With nDepth log storage, each data partition/shard is also signed and encrypted, so when the data is synchronized to the share in either case it's not possible to modify without making it useless (i.e. falsify or tamper). This makes it less of an issue to create a single "packaged" download, but that might still be something that's useful, kind of like a .bkf file on the ol' NTBackup. (The downside of that is that your downstream backup provider ALSO has to back up a huge file that has changed, so many smaller files may still be better)


          With the current implemented database archive (archiveconfig), we were going for "make me a copy of what's there in case I need it later" (whether that's disaster recovery or archival purposes), but it's not very intelligent, and it is very large (and since the data's already relatively well-compressed, the compression takes a while and doesn't have a HUGE gain, like 10:1 or anything - in your case just under 2:1). Even though only a portion of your data changes from day to day, we still archive the entire thing.


          The other thing we've discussed is to make it easier to retrieve the archive and use it for search or reporting, or to access remotely as a "deep storage" mechanism like I mentioned above. There's basically 2 purposes for an archive (disaster recovery and long term storage) which would imply whether you intend to use it again or not, even as a "just in case".


          Again, thanks for your feedback, if you have any additional thoughts (or anyone else wants to add anything), keep it coming! Archiving is a really useful topic.

            • Re: FEATURE REQUEST - Add options for archive compression

              I second this.  it would be nice to have options to try to speed this up.  I'm new to LEM, but last time I tried doing an archive, I couldn't manage via the console and it was all day before I reboot the system to kill it so I could work in it.

              • Re: FEATURE REQUEST - Add options for archive compression






                I am hopeful you will help me.  I am new to LEM and I need a little bit information about archiveconfig.  Reading various Thwack post and Knowledge Base Articles, I started a monthly archiveconfig and weekly backupconfig.  While I have found a recovery procedure for the weekly backupconfig, I am not finding a recovery procedure for the monthly archiveconfg. 




                In summary, I read post mentioning disaster recovery.  While I understand archiveconfig backs up “the normalized alert database on your LEM appliance”,  I am not sure I can restore this information back into LEM.






                Is there a procedure for importing the archiveconfig?


                In the event LEM crashes and I want to rebuilt, do I only need to use the import command for the backupconfig files? 




                Thank you,