Kiwi Syslog Service hanging

mhaley​

It sounds like you are in the correct place. I would definitely open a support case, if you haven't already, and let them take a look at everything.

(Submit a Ticket | SolarWinds Customer Portal )

You can check some simple/basic Kiwi diagnostics by clicking "Manage", then "Debug Options", then "Get diagnostic information".

The diagnostics will show you some basic stats for the server itself, top talkers, dns stats, static host entries, and various message stats. If you scroll down, towards the bottom half of the report, you should find some stats relating to message buffers. I would check those first, and see how if you have any overflow messages, and what percentage free is available. I have had numerous different issues cause the service to stop. While I have not performed the same actions you have, the last time I ran into this issue, I was simply adding a new rule. Another time, they narrowed it down to a database issue, as I had several rules dumping data into different tables in the same database.

What Kiwi version are you running?

How many rules do you have built?

Do you have any rules dumping data to a database?

How many message per hour does your indicator reach when it stops?

I wish I had more experience to offer you, but I'm pretty low level myself... Just trying to fill the void until the real help arrives.

Thank you,

-Will

To add to what Will said, check the error log in the Program Files (x86)\syslogd folder. My initial thought is that you are overloading the service causing it to crash.

what does the MPH show when it's frozen? If it's over 2 million that can cause problems. If the buffer is at 0% and/or your you're seeing overruns there is too much traffic.

We run into a similar issue, the only thing changed on our machine are windows updates i'm getting the following errors (allready made a support ticket also)

Please post the message per hour like kstone mentioned.

I get the same errors right when the logs stop being created.

2017-05-31 08:01:47 *** INTERNAL PROGRAM ERROR - Please contact http://www.kiwisyslog.com/support/ ***

2017-05-31 08:01:47 Service Version =9.6.1.6 | Error Number: 13 | Description: Type mismatch | Module Name: Syslogd.frm | Procedure Name: ParseMessageQueue | Line Number: 420 | Date and time: 5/31/2017 8:01:47 AM

Google led me to a Resolution that this was fixed in ver. 9.5.1, but I'm using ver. 9.6.

.

I am trying to open a ticket with Solarwinds.

I'll try to answer your questions.

What Kiwi version are you running?      9.6.

How many rules do you have built?      Not sure on this, I didn't build it.

Do you have any rules dumping data to a database?      I don't believe so.

How many message per hour does your indicator reach when it stops?      It's set to alert when it hits less than 10,000/hr.

Below are the logs that were in progress when it stopped.

8 o'clock hour today

9 o'clock hour today

Buffer count seems to be ok, no overflows.

And the service has stopped again.

Although if I hit start, it tells me it's already running.

I have to stop then start.

Kiwi Syslog Server [Licensed] Version 9.6.1.6

///         Kiwi Syslog Server Statistics           ///
------------------------------------------------------------
Syslog period ending on:          Wed, 31 May 2017 10:52:43
Syslog Server started on:         Wed, 31 May 2017 09:03:43
Syslog Server uptime:             1 hour, 48 minutes
------------------------------------------------------------

+ Messages received - Total:                    0
+ Messages received - Last 24 hours:            0
+ Messages received - Since Midnight:           0
+ Messages received - Last hour:                0
+ Message queue overflow - Last hour:           0
+ Messages received - This hour:                0
+ Message queue overflow - This hour:           0
+ Messages per hour - Average Last 24 hours:   
+ Messages forwarded:                           0
+ Messages logged to disk:                      0

+ Errors - Logging to disk:                     0
+ Errors - Invalid priority tag:                0
+ Errors - No priority tag:                     0
+ Errors - Oversize message:                    0

+ Disk space remaining on drive D:              675908 MB

Message Buffer Information (Secure TCP)
=======================================
Message Queue Max Size: 10000000
Message Queue overflow: 0
Message Count:          0
Message Count Max:      0
Percentage free:        100

Message Buffer Information (UDP, TCP, SNMP)
===========================================
Message Count:          0
Message Count Max:      0

E-mail Buffer Information
==========================
Message Queue Max Size: 1000
Message Queue overflow: 0
Message Count:          0
Message Count Max:      1
Percentage free:        100

I had a similar problem except I couldn't get kiwi syslog server running on w2k12R2 server that was also a DC... we're going to move it to a windows 7 host instead as a solution.  I couldn't get the service to start and stay running at all.

It seems that some .net updates or something screwed up the compatibility , coworker found the main issue with the setup of the Kiwi Syslog Service Manager and updates our support ticket with this information. Now we are a week into this ticket and we didn't get a response anymore.

It seems that the DLL has changed with some dependencies. This causes issue with the manager.

To solve the issue with the manager, I can confirm that copying the DLL to the program folder itself solves the messages and errors for about 98% (as in everything works, but sometimes there is a new error that pops-up because of error handling (according to .NET explanation)).

2017-06-01 11:56:59       *** INTERNAL PROGRAM ERROR - Please contact http://www.kiwisyslog.com/support/ ***

2017-06-01 11:56:59       Service Version =9.6.1.6 | Error Number: 13 | Description: Type mismatch | Module Name: Syslogd.frm | Procedure Name: ParseMessageQueue | Line Number: 420 | Date and time: 1-6-2017 11:56:59

2017-06-01 11:59:28       *** INTERNAL PROGRAM ERROR - Please contact http://www.kiwisyslog.com/support/ ***

2017-06-01 11:59:28       Service Version =9.6.1.6 | Error Number: 13 | Description: Type mismatch | Module Name: Syslogd.frm | Procedure Name: ParseMessageQueue | Line Number: 420 | Date and time: 1-6-2017 11:59:28

2017-06-01 12:05:25       *** INTERNAL PROGRAM ERROR - Please contact http://www.kiwisyslog.com/support/ ***

2017-06-01 12:05:25 Manager Version = 9.6.1.6 | Error Number: 440 | Description: Automation error | Module Name: Syslogd.frm | Procedure Name: UpdateGridDisplay | Line Number: 30 | Date and time: 1-6-2017 12:05:25

In regards to the Syslog service itself kicking the bucket, we’ve found out that the cause of the service hanging itself is UDP and TCP syslog on one and the same port (514), will kill the service. If we try the same with an installation we’ve not yet updated to this newest release, it doesn’t crash, but with the newest release it crashes.

For now it seems like we’ve reached an all new record of the service and manager running for 10 minutes without crashing

mhaley​ & martijng​

If you are having trouble with support, perhaps DanielleH​, jeff.stewart​​, or bkyle can assist with getting some momentum going on those tickets. Each of you may benefit from requesting an escalation. Also, it might be a good idea to post your case # here, in case Danielle or Jeff need to help push the tickets to attention.

I'm glad you're making progress, but sorry to hear support hasn't been able to fix it, yet.

Thank you,

-Will