10 Replies Latest reply: Dec 12, 2013 5:52 PM by byrona RSS

Distributed architecture?

byrona

Does LEM support any form of distributed architecture that would allow you to have collectors at different locations and/or networks where the data is then rolled up into a single interface for visualization, searching and reporting?

 
  • Re: Distributed architecture?
    curtisi

    Yes.  The solution would be deploying something like Kiwi Syslog Server in different network segments.  Devices in those segments would send their syslog data to their segment's Kiwi server.  The Kiwi machine would have the LEM Agent installed, which would take the data, normalize it, and send it on to the LEM for visualization, correlation and rules.  The LEM would show the Kiwi server(s) as nodes, and the Kiwi servers would see the individual source hardware.

    • Re: Distributed architecture?
      byrona

      Hrm, I actually thought about this a while back and proposed it to some folks here.  I feel silly for not thinking about it when I posted this request.

       

      With that being said, I still think there should be a distributed architecture all within the LEM product.

      • Re: Distributed architecture?
        curtisi

        The LEM code-base includes an option for stripping the Manager functions and converting the LEM to a syslog server.  This process launches an Agent on the syslog server and connects it back to the LEM Manager.  The benefit of this deployment is that the conversion also regenerates the product support key in such a way that it will run forever for free.  It also doesn't require as many resources (less memory) to run, since it's not doing any of the manager functions.  The LEM syslog server can log more events per day than Kiwi.

         

        (Some of) the gotchas:

        • Converting the LEM to a syslog server requires that you call support, as root access is needed to initiate the process
        • Converting the LEM to a syslog server is only an option if you have support on a LEM manager
        • The LEM syslog server will send every event on to the LEM manager, which may result in undesirable amounts of traffic

         

        The Kiwi solution has a couple of things it can do that LEM can't: for example, you can set rules in Kiwi to filter which events get forwarded to the LEM Manager.  This means that the amount of chatter between Kiwi and LEM can be controlled, and if a lot of events you don't care about are occurring in a network segment, you can stop them at Kiwi instead of sending them on to the LEM.

        • Re: Distributed architecture?
          byrona

          Thanks for the additional info.  I have used Kiwi before and am familiar with it's capabilities, it's certainly a good option for the cost.  I could certainly use Kiwi as a form of remote collector that sends back to LEM and for the cost it would make sense.

        • Re: Distributed architecture?
          byrona

          What Kiwi doesn't give me is the agent functionality, I would like to see a distributed architecture that leverages the agent capabilities.  For example, if I had the ability to set a flag that would turn a single agent at a site as the relay agent that all of the other agents at that site talked to and then that relay agent would communicate back to the manager.

           

          The other option would be a master of masters design where you have LEM systems that all report to a higher up master system.  My understanding is that this partially exists now but not with full functionality.

          • Re: Distributed architecture?
            curtisi

            I think with the Agents, the more realistic scenario is multiple LEMs, and then connecting to those LEMs via the console.  The console would let you monitor and build rules, groups and users on any connected LEM.

          • Re: Distributed architecture?
            nicole pauls

            As it is now, think of it as scaling vertically, but not horizontally. You can build a stack to distribute functions, but you can't build multiple stacks that work together.

             

            Single virtual appliance:

            • Collect syslog data and process with effective 'agent'
            • Consolidate data from all agents
            • Correlate consolidated data
            • Store consoldiated data
            • Connect and display/search/report on data

             

            "Stack" model:

            • Virtual Appliance 1:
              • Collect syslog data and process with agent (you can also use Kiwi on a Windows system, syslog-ng on Linux, etc)
            • Virtual Appliance 2
              • Consolidate data from all agents
              • Correlate consolidated data
              • Connect and display/search/report on data
            • Virtual Appliance 3
              • Store consolidated data (you could also have the original log storage here or on its own appliance)

             

            The distributed model could have multiple collection points, multiple correlation points that are either "rolled up" (local system A collects and processes for one network, remote system B gets critical events forwarded to it).

             

            LEM also doesn't have the concept of collectors, which you do see with some other SIEM architectures, kind of a delegation of the collection and temporary storage like you can do with a syslog server but for agents. We are looking at building agentless Windows Event Log connectors, which would serve that purpose for data collection from the Event Logs at least (one system would reach out to other systems and aggregate logs to pass back to the appliance using that single system's agent without putting agents on all other systems).  Downside is no active responses or USB Defender remotely (or things like FIM in the future) and you're at the mercy of Windows APIs for encryption and communication, but in some environments it's useful.

             

            Usually these architectures are used to help scale, so we continually look at what our real cap is within the single appliance approach and at what point we need to break the model up.

             

            HTH

            • Re: Distributed architecture?
              byrona

              I would agree that it supports a vertical scaling model but not horizontal.  My question about this is less a concern for scaling and more an interest in distribution.  We have some remote sites where I would like to have all of the nodes (agents) at that site talk to a single relay that then communicates back to the LEM appliance.  I know I am probably the only weirdo out there that would use this but figured I would ask about it.

              • Re: Distributed architecture?
                nicole pauls

                It's okay, I'm used to it.

                 

                The same collector/aggregator request applies to low bandwidth sites, especially when you talk about having to send syslog over the WAN which is unencrypted and usually UDP to boot. In that case we recommend a syslog server, but it would still be nice to aggregate the agent data as well, then when the link gets severed it's one thing connecting to the Mother Ship, not 100.

                • Re: Distributed architecture?
                  byrona

                  Our patching product we use has an option to turn it's agent into a relay, it's just a check box you click on the agent.  Then you are able to point other agents at that system.  It's a super simple and works really well.  I was thinking something like that would work really well with LEM also.