5 Replies Latest reply on Jan 3, 2012 4:33 PM by phil3

    Store events from specfic nodes in a different database?

    qle

      I don't think this is possible but I'll ask to make sure. Is it possible to segregate collected events from specific nodes into a separate database?

        • Re: Store events from specfic nodes in a different database?
          phil3

          Hello again, Quang.

          It looks like this one's going to be a "Yes, but..." kind of answer. :)

          You basically have two databases available to you with LEM: one for alert, or "normalized," data, and one for original, or "raw," log messages. The former is used for your filters and rules, and is also available for reporting and nDepth searches. The latter is only available for nDepth searches (which can also be used for ad hoc reporting). This can be useful if you have one group of devices you want to actively monitor, and another group that you just want to aggregate logs for; but it can be difficult to draw this line in most environments.

          The neat thing about LEM (as opposed to the former TriGeo SIM iteration of this product) is that you can have as many external storage virtual appliances as you want. Only one VM can be designated as the Manager, but you can have another one for your alert DB and yet another one for your "raw" DB. Just deploy the original OVA like you did for your initial configuration and from there there's just a few tweaks that will need to happen on the back end to make it all work together.

          For more information about storing and searching "raw" data in a separate database, check out these KB articles:

          Thanks for the question.

            • Re: Store events from specfic nodes in a different database?
              qle

              Phil,

              As always, your detailed responses are appreciated.  I remember coming across this group of KB articles and wondered what purpose it served.

              Obviously, I have a few questions about this implementation.

              1. If I'm understanding this correctly, I can, as an option, designate a separate virtual appliance as storage for the "raw" log messages?
              2. Therefore, I could have at most three VM appliances as part of the same deployment: one management VM, one raw log VM and one alert log VM?
              3. Is it fair to assume that all licensing is handled by the management VM and no licenses would be required for the other two?
              4. I know one of the benefits of LEM's agent-based solution is bandwidth efficiency. For those particular tools that are configured with an output for Alert, nDepth, will I lose this benefit?
                • Re: Store events from specfic nodes in a different database?
                  phil3


                  If I'm understanding this correctly, I can, as an option, designate a separate virtual appliance as storage for the "raw" log messages?

                   



                  Yes.

                   



                  Therefore, I could have at most three VM appliances as part of the same deployment: one management VM, one raw log VM and one alert log VM?

                   

                   



                  The main restriction to be aware of in this regard is that you can only have one "Manager" appliance. In addition to database appliances, though, you can also have appliances to use as network sensors or logging servers. Splitting your load up in this fashion can sometimes help with Manager performance - especially if you have a lot of one particular type of data.

                   





                  Is it fair to assume that all licensing is handled by the management VM and no licenses would be required for the other two?

                   

                   



                  Each additional appliance, and all sources logging to it, consumes a node license. Regarding other licensing, this is where the "only one Manager" stipulation comes into play.

                   



                  I know one of the benefits of LEM's agent-based solution is bandwidth efficiency. For those particular tools that are configured with an output for Alert, nDepth, will I lose this benefit?

                     



                    I can't imagine any significant loss of benefit in that regard, but that would depend at least slightly on any bandwidth restrictions you might already be limited by in your environment. The big concern here would be with database size. That's where the additional appliances come into play, since pointing any connector to both the Alert and nDepth DBs essentially duplicates your storage requirement for that device.

                    Thanks again for all your thoughtful questions.

                      • Re: Store events from specfic nodes in a different database?
                        qle


                        The main restriction to be aware of in this regard is that you can only have one "Manager" appliance. In addition to database appliances, though, you can also have appliances to use as network sensors or logging servers. Splitting your load up in this fashion can sometimes help with Manager performance - especially if you have a lot of one particular type of data.

                         



                        Can you elaborate more on this? How would one go about configuring appliances as network sensors or logging servers? Those KB articles seem to only reference a raw log (nDepth) appliance.

                         



                        Each additional appliance, and all sources logging to it, consumes a node license. Regarding other licensing, this is where the "only one Manager" stipulation comes into play.

                         



                        Ok, I understand that each additional appliance would require a separate (single?) license. However, if I reconfigure a tool instance to output to both Alert and nDepth and have a separate nDepth appliance, how would the licensing work? Is it only one license, on the appliance that is the "collecting" the logs/events or is it two, one for the collection and one for redirecting/forwarding for storage on the nDepth appliance?

                         

                         

                        I can't imagine any significant loss of benefit in that regard, but that would depend at least slightly on any bandwidth restrictions you might already be limited by in your environment. The big concern here would be with database size. That's where the additional appliances come into play, since pointing any connector to both the Alert and nDepth DBs essentially duplicates your storage requirement for that device.

                         



                        I understand. The main reason for this question was the fact that the agent is bandwidth efficient. This allowed us to "reliably" collect logs from remote offices. Would I be compromising this by duplicating the traffic generated by each remote node?

                          • Re: Store events from specfic nodes in a different database?
                            phil3


                            Can you elaborate more on this? How would one go about configuring appliances as network sensors or logging servers? Those KB articles seem to only reference a raw log (nDepth) appliance.

                             



                            The KB articles cited above refer to the processes for configuring the nDepth database, not a separate appliance. The Manager and both databases can all reside on a single appliance, but you can also deploy multiple appliances as discussed above. If you want to deploy any additional appliances -- regardless of their role -- you'll want to contact Support because all of the processes to do so require escalated permissions that aren't available directly to customers.

                             



                            Ok, I understand that each additional appliance would require a separate (single?) license. However, if I reconfigure a tool instance to output to both Alert and nDepth and have a separate nDepth appliance, how would the licensing work? Is it only one license, on the appliance that is the "collecting" the logs/events or is it two, one for the collection and one for redirecting/forwarding for storage on the nDepth appliance?

                             

                             



                            A single device should always only equal a single licensed node, regardless of how many databases/servers it's pointing to. If you see something different in your LEM Console, contact Support.

                             



                             

                            I understand. The main reason for this question was the fact that the agent is bandwidth efficient. This allowed us to "reliably" collect logs from remote offices. Would I be compromising this by duplicating the traffic generated by each remote node?

                             



                            In either case, your LEM Agents will still be sending events to the Manager as they come in, so there will rarely be any big chunks of data flowing from the Agents to hog bandwidth. This real-time data flow is where the efficiency and reliability comes from. Other SIEM products, on the other hand, collect the data and then send it in batches -- that's where you'll really start to notice bandwidth implications.

                            Furthermore, whether a connector is sending data to just the alert DB or both the alert and "raw" DBs, the data is compressed before being sent to the Manager. That's another source of bandwidth efficiency that's not greatly compromised when sending to multiple DBs.

                            I hope these answers are helpful.