1 Reply Latest reply on Nov 8, 2016 9:13 AM by curtisi

    unstructured app logs in LEM?


      hi. About to try an evaluation of LEM, but wanted to know if it's capable of handling unstructured logs as well as standardized system logs. For example, we would have an application log that is generated from a custom or home-grown application. It might contain timestamps, specific data payloads as well as instrumentation data around transactions. Could be single or multi-line. Regex would typically be the way to parse the data. Is that something that LEM does out of the box? Or are there ways to customize? Marketing material is limited in details.


      Here's a quick sample of something we might want to parse and report on, with correlation with other logs across the stack.

      0000183E-8E71-4DE0-92C6-B0EFAA096C0E|93075243|5CFF8C57-FFD4-492B-A2E1-3A0A3E583E7E|XSQL.POSTSAVE|EXTM|Sep 21 2016  6:00:56:406PM|Sep 21 2016  6:00:56:483PM|ceraxmgr.cpp|652|CER_EXIT_MGR::CallExit|ceraxmgr.cpp|654|CER_EXIT_MGR::CallExit|E|1|Jan  1 1753 12:00:00:000AM


      Is the system capable of parsing this type of log using regex or something similar to know that fields 6 and 7 are timestamps?


      Similarly, is it capable of reading something like nmon data from AIX? nmon is notoriously verbose and hard to parse.



        • Re: unstructured app logs in LEM?

          AIX NMON Data (and other)

          LEM can take data from AIX syslog, and we have an AIX Agent for some IBM platforms.  For others, you'll need to look at a third-party agent like Patrick Townsend.  I don't see any reader for NMON in particular.


          Unstructured Data

          LEM parses data using "connectors," which is our term for "an XML file loaded with regex to parse and classify specific vendor or product log types and formats."  We include a lot of connectors out-of-the-box, and as part of maintenance as a LEM customer you can request new connectors or improvements to existing connectors.  You log line (which is terrible, btw: if this is a custom app, can you make it use the syslog RFC?)  could be parsed, but when requesting a connector for a homebrew app (or anything) I think it's safe to say we consider the following:


          1. Is this a request that will benefit more than a single customer? (ie, if you noticed that your Cisco device was sending a log we didn't normalize, we'd want to help you out because it helps the whole customer base.  Or, if you buy a new device or software from Wodget Corp, but it looks like Wodget Corp will eventually sell to thousands of people, we want to add Wodget Corp to our library)
          2. Is this a request that we can use pre-existing work to complete?  If your logs follow a format we already have general readers for (like syslog or the Windows Event Log format), it makes a new connector a lot easier for us to build.  If not, that can cause a longer dev cycle
          3. And this one is an open secret: Solarwinds is in this business to make money, so if this connector is holding up the sale or renewal of a million dollar LEM license, that helps it's priority


          First thing based on that: is this app something that will ever be in the wild?  Or is this just an internal application?


          The other red-flag in your example is 'it might be single or multi-line,' and I can tell you that logs that jump back and forth between formats are a nightmare to normalize for LEM, especially if (as I suspect) your multi-line logs don't include indexes to link the multiple line together.