NIST EMAP protocol

joesecgeek

Does Solar winds plan on developing or complying to the new NIST EMAP standard?  An attempt to standardize log records by content and type.  You can look here; 

http://chuvakin.blogspot.com/2011/06/nist-emap-out.html

and here

http://scap.nist.gov/emap/

 

Thanks.  I like the product, but this is a showstopper for me. 

FormerMember

As a part of normalizing data, we do parse and translate events to a taxonomy format, which is passed through the system. (In parallel, if enabled, the original source data is also passed on for storage.) After it's parsed, this taxonomy format is the actual data passed through the LEM engine for monitoring, correlation, and storage. EMAP effectively forces a standardization of that normalized format into the NIST standard formats (endorsed specifications like CEE/OEEL/CERE), for both correlation and storage. It basically highlights many core features of SIEM - normalization to a common format, parsing, storage - and standardizes them so that all SIEMs would process and store their data using the same methodology. That stuff is not exactly SIEM secret sauce, though when you talk about storage and correlation, you do intersect with secret sauce territory a bit, as many SIEMs have specifically structured their data to make it fast to store, search, and report, and in the case of some SIEM products including LEM correlate in memory in near-real-time.

In the full EMAP standard, the log/data parsing would begin with OEEL rules that converts to CEE. CEE would be used to express the data for processing and storage - i.e. if data was not already received in CEE, we would translate it to CEE and normalize to our expected format, then pass on to the other parts of the engine and store that way. CERE would be used to describe the correlation rules, and anything actionable would be expressed as ARF. 

The benefits of that in the LEM world are really only in the externalization of data - reporting, searching, external alerting, etc - as everything else relating to the parsing and correlating is internal to LEM and not exposed to the customer at that level. For example, our correlation engine is exposed to the Console via a drag and drop interface, and the actual language of the rules themselves are never exposed - reducing the complexity of working with the correlation rules but also making EMAP (CERE in this case) a non-issue from a customer usability/interactivity perspective. Our connectors do parse the data, but this process is also not externalized to the customer, making EMAP (OEEL) a non-issue from the customer usability/interactivity perspective as well.

You can imagine the scope of implementing something like the complete EMAP standard across the foundation of a product that has been in the field for over 10 years. We have experience with receiving data in CEE and continue to watch that effort, but it has not yet become foundational to the way we work with data. I would expect the same pattern with the other EMAP standards (OEEL/CERE).

As we evolve areas of LEM we look to standards to see if they are the best way to accomplish our goal. If we continue to evolve our parsing or rules engines, for example, we look at whether OEEL/CERE would accomplish what we want to accomplish in the best way to satisfy our (customer) requirements. The latest EMAP stuff did just come out in June, and prior to that as with CEE things did move along slowly and it is always hard to say what the adoption and/or customer expectation rate will be. We like to invest our resources in high customer value features.

Long story short: no, we don't, but I'm not aware that any product does today. We look to customer feedback and as there's a growing need, we will implement those standards. I appreciate the feedback and will be sure to document your request so we can track further requests.

If you have questions about how EMAP standards would apply to LEM based on what I've said here, feel free to follow up. Thanks!