6 Replies Latest reply on Apr 20, 2015 2:09 PM by nicole pauls

    Extend data retention


      I'm aware that data retention is determined by amount of storage allocated to LEM. Running the Database Maintenance Report, I see that we're currently sitting at a little over a 6-month retention. We're interested in extended that further out so I'm looking at various options to present to management. I'm aware of the following two options:


      1. The easiest option would be to allocate additional storage to LEM by increasing its size.
      2. I'm also aware that reducing the amount of events/logs sent to LEM, will, in effect, extend retention as well.


      However, I've been reading about separate nDepth and/or database appliances. Would this help us accomplish this?

        • Re: Extend data retention

          The separate appliances option was something Trigeo did in the days before virtualization.  When you were constrained by a physical chassis and the number of platters on a spindle, sometimes the only way to expand the appliance was with another box.


          Now that virtualization, SANs and LUNs are a thing, the need for additional appliances has been largely replaced, and with the virtual appliance it's not a supported deployment.  Besides, why have two VMs chewing up resources and adding latency to your logging when one beefy VM will do the job?

            • Re: Extend data retention

              curtisi, thanks for the explanation. That makes sense.


              What about the backup archive. Could that be a mechanism to extend retention? I'm aware that you can't merge the archive back into the live data. So is it possible to search the archive using the console? If so, how would one do that. I'm having trouble finding a knowledgebase article on this.


              Also, what would be both the performance and storage impact be compared to simply expanding LEM's volume?

                • Re: Extend data retention

                  Yes, it could expand retention.  No, you can't search the archive from the console of your live LEM.


                  The way this works would be:

                  1. Setup the ARCHIVECONFIG command on your production LEM.  This copies DB data to another server in a proprietary and encrypted format.
                  2. Time passes...You need data from the past that is no longer on the production LEM
                  3. Download the LEM virtual appliance from the website and spin up an eval LEM
                    • The LEM only needs a license to receive new data from hosts.  Since this eval LEM will never need to get new data, it will never need a license.  You can access the console, run nDepth searches and run Reports forever if you want
                  4. Call support, and have them import your archived database partitions into the eval LEM
                  5. Run your reports and searches, using the eval as a data warehouse
                  6. When you're done (you got what you needed, the auditors are happy) blow the eval LEM away or keep it as a historical archive


                  Because this eval LEM won't be doing rules and correlations or processing new data, you could probably run it at something below minimum spec to conserve resources, but I'd check with support on that.


                  Performance - There is no measurable impact to the LEM.  Searching a week of data from a month ago will be as fast as searching a week of data from a year or five years ago if the LEM disk is large enough.

                  Storage - Expanding the LEM's volume will cause it to use more storage...?  I don't think I understand this question so much.

                • Re: Extend data retention
                  nicole pauls

                  We can actually deploy a dedicated database appliance if it's necessary, but we've found in our experience that's rarely the case. I can't say we've had a customer go this route in a very long time, since the all in one appliance works so well.