This discussion has been locked. The information referenced herein may be inaccurate due to age, software updates, or external references.
You can no longer post new replies to this discussion. If you have a similar question you can start a new discussion in this forum.

Syslod Services Stopped

I discovered that one of the server running the Kiwi Syslog has its syslogd service stopped and not taking in logs from other servers. Checking Services.msc 'Kiwi Syslog Server' is still running while in Kiwi Syslog Service Manager showed 'The Kiwi Syslogd Service is stopped'. Checking Event Viewer found that the service stopped.syslog stop.JPG

I've asked the server team if there was any activities during that period, there were SEP was being installed on all servers recently and scheduled full scanning were taking place. I've asked the server team to temporarily disabled scheduled full scanning and so far the service is running without any issues, however I do not want to conclude that SEP is the cause of the service to stop just yet. On average Kiwi Syslog pipes 1800K -  1900K MPH and at times can peak up to 2500K MPH. If it helps this is the hardware specs of the server.

Xeon E5-2680 2.40GHz

4GB RAM

Windows 2012 R2 64bit

1.5TB Storage

Does upgrading the RAM helps? Is there any other possible cause for the service stoppage?

Any and all help is greatly appreciated

  • Additional RAM is unlikely to help Kiwi much but might. Watch the Task Manager for a while to see what the peak usage it. Unless Kiwi is queuing messages it doesn't use a lot of RAM, we average less than 250MB at about 500k MPH.  It's also a 32bit app so its total usage it limited.  More than 4gb would give the OS some space to operate which may improve performance overall.

    1.8M MPH is very close to the upper rated limit of the syslog engine(2M MPH)  If you are running rules for parsing, writing logs to disk, alerts, etc, those will all reduce that upper limit. 

    Have you looked at the Kiwi error log(C:\program files (x86)\syslogd\errorlog.txt) to see what is being recorded when the app crashes?

  • I never got the chance to look at the error logs (services stopped on the weekend), by the time I got back the errors have been over written by the most current dates and there is nothing about 7th Dec. Is there a way to archive the error logs?

    Update: I've been monitoring over this week. I've disabled SEP scheduled full scan on the server. There is no service stoppage. Will monitor this week with SEP scheduled full scan on.

  • Antivirus can definitely be an issue...

    You can archive error logs.  You need to create a scheduled task in Kiwi to look for C:\program files (x86)\Syslogd\errorlog0.txt.  The error log rotates at 1mb and keeps 1 older version.  I have an 'Archive' task that checks for that file hourly and if found copies it to a new folder.  I also have another 'Cleanup' task that deletes any files in the archive folder that are more than 30 days old. 

  • Syslogd stopped again.

    I took a look at the error logs and this is what I got

    2019-12-19 02:13:03       Log to file action - Error: Win32File Object [45600] Unknown error.

    2019-12-19 02:13:03       Log to file action - Error: FlushCacheLines <Encoding_Failed> - File: D:\Syslogd\Logs\Syslog-[REDACTED]-2019-12-19.txt

    2019-12-19 02:13:03       Log to file action - Error: Win32File Object [45600] Unknown error.

    2019-12-19 02:13:03       Log to file action - Error: FlushCacheLines <Encoding_Failed> - File: D:\Syslogd\Logs\Syslog-[REDACTED]-2019-12-19.txt

    2019-12-19 02:13:03       Log to file action - Error: Win32File Object [45600] Unknown error.

    2019-12-19 02:13:03       Log to file action - Error: FlushCacheLines <Encoding_Failed> - File: D:\Syslogd\Logs\Syslog-[REDACTED]-2019-12-19.txt

    2019-12-19 02:13:03       Log to file action - Error: Win32File Object [45600] Unknown error.

    2019-12-19 02:13:03       Log to file action - Error: FlushCacheLines <Encoding_Failed> - File: D:\Syslogd\Logs\Syslog-[REDACTED]-2019-12-19.txt

    2019-12-19 02:13:03       *** INTERNAL PROGRAM ERROR - Please contact http://www.kiwisyslog.com/support/ ***

    2019-12-19 02:13:03 Service Version =9.5.2.5 | Error Number: 14 | Description: Out of string space | Module Name: FileStore.cls | Procedure Name: Add | Line Number: 40 | Date and time: 12/19/2019 2:13:03 AM

    2019-12-19 02:13:03       Log to file action - Error: Win32File Object [45600] Unknown error.
    2019-12-19 02:13:03       Log to file action - Error: FlushCacheLines <Encoding_Failed> - File: D:\Syslogd\Logs\Syslog-HQ-GSAN-FW-2019-12-19.txt
  • Looks like too much traffic and the server is getting overrun and crashing. 
    Do you have a lot of rules? How many log to files? How many have multiple filter conditions?

    As I mentioned previously you’re very close to the capacity limit of Kiwi Syslog.  There are some tuning tips and tricks that can help but reducing the ingestion amount is probably the only solution.

  • I have 35 rules, all of the rule has log to files.

    Filter conditions are IPv4 Simple to look for specific IPs and String Simple "Security" (for security logs) and "Application" (for SQL database logs)

    120 servers, 73 network appliances, 14 SQL DBs

  • If you don't have them already, add a 'Stop Processing' action to the end of each rule that doesn't need to match any additional rules.  This keeps the message from passing to the next rule and being processed again.  It can dramatically decrease the the processing the rules engine has to do.

    With that, make sure the busiest rules are at the top of the list.