This discussion has been locked. The information referenced herein may be inaccurate due to age, software updates, or external references.
You can no longer post new replies to this discussion. If you have a similar question you can start a new discussion in this forum.

How do I exclude or filter out alerts?

The issue I've always had with SIEM systems is that they aggregate all of my logs and leave me with a sea of data that is far too expansive to swallow. I'm trialing LEM now in hopes of finding a product that might defy the odds.

As such, is there a way in LEM to auto-remove or filter out (not create a filter to see) certain superfluous events?

In my case, I've set my VMware ESXi 5 hosts to syslog to my LEM appliance, and I'm getting inundated with events about "Power policy is unset", which seems to just be noise in the vpxa/hostd logs about VM details (which don't even warrant a log entry in the vSphere Client events list). In 8 minutes, I get 1000 of these events from just 1 host. As you might understand, I'd like to omit these if possible.

If that isn't how LEM is supposed to be configured, can someone explain the proper way to handle extra data like this? I'm hoping to use LEM to harvest "CatchAll" log files from my Kiwi Syslog Server, so I expect I'll start finding plenty of data I don't wish to retain.

Thanks,

Chris

  • Hi, Chris.

    First of all, which filter(s) are you using to view these events? Are you just looking at the All Alerts filter right now, or have you created one for those ESXi hosts?

    In either case, you can add a line to the filter logic to omit these superfluous events. Just identify something special about the alert (something in the EventInfo or ProviderSID fields, for example) and then add a line similar to AlertName.FieldName != SpecialValue. You can also use wildcard characters to simplify or generalize the "special value" portion of the condition.

    The default User Logons (interactive) filter is a good example of how to do this. In this case, the logic is:

         UserLogon.LogonType != [blank]

         UserLogon.LogonType != Windows: Network

    With this logic, we see all of the UserLogon alerts except those with a blank LogonType field or that contain the exact value "Windows: Network" in that field. Similarly, if we wanted to simplify the latter condition, we could have entered *Network* into the text constant field, using the asterisks as wildcard characters.

    For more information about working with filters, check out these KB articles:

    Hope this helps.

    Phil

  • Hey Phil,

    While replying, I wrote a lot about how this and that didn't work and even how one of those KBs was wrong, but eventually started to figure out a few of the caveats (i.e. "Any Alerts" under "Alert Groups" -- very important or else it'll filter out more than you expect). I'm far from loving things, but it's at least a step in the right direction.

    From a design theory point of view, all of the alerts/events are still being accumulated in the DB, right? My filters just hide (as opposed to remove/delete) what I don't want to see, even from "All Alerts"). Is that correct? If so, is the Monitor tab and Filters list just a cache of recent events then? I notice when I change a filter and it clears out events (or floods the log, causing it to roll over the 1000 quickly), the prior events adding up to 1000 don't come back.

    Just trying to figure out the methodology and how the data is stored/presented, so I can build upon it. My primary hope is to provide visibility into the events that currently add up to around 220GB of hourly, per-device text file syslogs on a Kiwi server.

    Along those lines, I'm also trying to solve for the scenario in my other LEM thread, How do I harvest Windows syslogs from Kiwi? If you have any tips there, I'd welcome them, too. Concept/theory input is helpful, just the same.

    Thanks,

    Chris

  • cmgurley wrote:

    I wrote a lot about how this and that didn't work and even how one of those KBs was wrong, but eventually started to figure out a few of the caveats (i.e. "Any Alerts" under "Alert Groups" -- very important or else it'll filter out more than you expect). I'm far from loving things, but it's at least a step in the right direction.

    If there's anything specific you'd like us to tweak about any of the articles you use, please let us know. There's a "Comments" section for each article, and we check those weekly.

    cmgurley wrote:

    From a design theory point of view, all of the alerts/events are still being accumulated in the DB, right? My filters just hide (as opposed to remove/delete) what I don't want to see, even from "All Alerts"). Is that correct?

    Yes, you're correct that the filters just hide what you don't want to see. All of the alerts are stored on the DB unless you explicitly exclude them using Alert Distribution Policy. We don't recommend excluding any alerts from being saved, however, since you have to do so by alert name, which can encompass more than just the alerts you're currently seeing.

    cmgurley wrote:

    If so, is the Monitor tab and Filters list just a cache of recent events then? I notice when I change a filter and it clears out events (or floods the log, causing it to roll over the 1000 quickly), the prior events adding up to 1000 don't come back.

    This is also correct -- the Monitor view only shows real-time data and rotates that data after a certain number of alerts. You can increase this number to up to 2,000 alerts per filter in Filter Creation if you want. If you want to see historical data for your filters in the console, you can send any filter to nDepth as a search by selecting the filter and then selecting Send to nDepth from the gear menu at the top of the Filters pane.

    Here are a few more articles relevant to this discussion:

    The first one isn't directly related to your specific situation, but the concept should be helpful, just as my previous filter example (hopefully) was.

    I'll check out your Kiwi post and respond there if I can be of any help.

    Thanks for checking out LEM and for all of your helpful feedback.

    Phil

  • Thanks, Phil.

    I think that about wraps up this thread. I'll aim to engage with the pre-sales team next week once I return from a trip.

    ~Chris