What We're Working On - Kiwi Syslog

Brandon

Two things I think would be nice to have with Kiwi...

• Reporting
• Web Accss Performance Improvements
  • Maybe make it run under IIS

Elaborate on reporting?

I assume IIS to it can handle scale/volume/response time better?

Yes, I suggested IIS with the hopes that it could possibly help with the scale/volume/response times.

With reporting I would like something that can look back and historical logs and create reports on specified criteria.  The problem right now is that the logs are stored off to text files which can become large and Notepad has issues with that causing me to need additional tools to manually comb through mounds of text lines to find anything useful.  I have the option to send this data to a database but that seems to significantly impact the overall speed at which Kiwi can process logs and I am still left to write my own manual queries and use some additional tool such as SSRS for the reporting side.

Having reporting functionality built directly into the product would be nice so I don't have to build out several different components just to make it possible.

I am actually right in the middle of a customer engagement where we are looking at providing a log management service supporting a HIPPA environment and coming up with a good solution is not a easy as one would think even with the commercial options that are available.

At what point do you think you pivot over to LEM?  Since Kiwi is SNMP, Syslog and Windows Event only, that only partially covers you for HIPPA

I can't switch to LEM.  We just finished evaluating it for this customer engagement and it's not flexible enough.  Because it uses connectors to normalize data it isn't designed to look at any potential syslog which would include custom applications logging to syslog.  We need to be able to potentially look at any and all syslogs.

Ultimately we would like to build and offer an enterprise log management service; preferably using SolarWinds products because we really like you guys.  To do this we would need flexibility.  We would like to say that we expose a syslog facility and any logs you send to us via syslog we can process and manage.  Having the added agent ability that LEM has to get logs that don't send to syslog such as IIS would be really nice as well.  I was really hoping LEM would be our solution but it wasn't .

I noted a possible feature for LEM that would provide the additional flexibility...

Provide a development environment with a documented Regular Expression to allow users to create their own LEM connectors for Syslog.  I picture it something like UnDP tool for NPM.

The capability of logging to an SQL DB in kiwi can solve all of your reporting needs. I know that log analisys would be an awsome feature in Kiwi, but that's what the more expensive solutions are for and Kiwi is already very powerful and configurable.(Just have to watch your script execution times for custom parsing and data collection).

What I did for our ASA logs is created a custom DB format, log to an SQL DB and run reports/setup views etc... We also have tools like Crystal Reports available for the more clerical folks that need to look at logs for things other than our ASA.

I don't deny that it's already a powerful and configurable tool; however, I don't think that justifies product complacency and lack of continuous improvement.  SolarWinds has always been about community driven continuous product improvement (which is one of the reasons we heavily utilize their tools) and reporting is a common feature amongst Syslog servers, even some of the free ones.  Of course creating my own reporting solution as you have done is always an option; however, this is a feature I would like to see included even if it required me purchasing a more expensive edition of the software.

On the other hand, if SolarWinds had decided that reporting is not going to be part of Kiwi and notes this as one of the differences between it and say their more robust higher end LEM product then I am left with a decision to make on our end.

That's a valid point, and like I said, configurable log analysis and reporting would definitely be helpful in Kiwi. But it does have the capability as an entry point to those types of features. I wouldn't think you would get any robust reporting features though if you're wanting to report from ASCII text logs. I would think that you could currently use Kiwi's Rules and filter options to tailor your log files to your preference and get what you want out of it without even necessarily needing to log to a DB. What I mean, is you might be able to get Kiwi to do what you need it to do NOW without external tools or logging to a DB.

If it's a cost issue, you can also use SQL Server Express...

And I am not saying we won't do reporting, actually I have that on my list.  Now what you define as reporting and what I define as reporting in Kiwi we will have to discuss and debate cause that could go 20 different ways. 

Alot of this ultimately is going to boil down to the use cases or problems you are wanting to solve.  There is going to be a point with some problems or use cases that Kiwi will just not solve and a LEM is going to be the route to go and there is going to be some overlap as well, but like I said, alot is going to depend on use cases/problems.

For example.  Above saying IIS is how to solve the problem, but what problem are we solving?  Your problem was slow response and load times.  That is a requirement I need to pass to Dev.  Personally I am not a fan of IIS for Kiwi cause it has a more intensive setup and management for a tools like Kiwi.  For a few hundred dollar product, most customers want something self contained that is easy to install and maintain.  I don't want to have to go and find my Windows 2003 disk to install IIS.  Anyways you get my point.

We are going to continue to move Kiwi forward, but at the end of the day there is going to be a pivot point based on the use case on which direction or product you would use.