cancel
Showing results for 
Search instead for 
Did you mean: 
Create Post
Level 9

Syslogd_Service.exe crash - out of stack space

Jump to solution

I am evaluating Kiwi Syslogd to front-end and filter syslog traffic since we are having performance problems and service crashes using the NPM Syslog Service.  Here is the hardware platform:

HP DL385G7
2x AMD Opteron 6174 2.2GHz 12-core processors
32GB memory
RAID-1 for OS/Syslog
Windows Server 2008 R2 x64 Enterprise SP1

I installed Kiwi Syslogd and it ran for about an hour before it crashed with this failure:


Log Name:      Application
Source:        Application Error
Date:          3/15/2012 10:42:42 AM
Event ID:      1000
Task Category: (100)
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      *********
Description:
Faulting application name: Syslogd_Service.exe, version: 9.2.0.1, time stamp: 0x4d069c0f
Faulting module name: unknown, version: 0.0.0.0, time stamp: 0x00000000
Exception code: 0xc0000005
Fault offset: 0x0000000a
Faulting process id: 0x91d0
Faulting application start time: 0x01cd02c944ab6d53
Faulting application path: C:\Program Files (x86)\Syslogd\Syslogd_Service.exe
Faulting module path: unknown
Report Id: 43e40d87-6ec6-11e1-a52f-3cd92b024752
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Application Error" />
    <EventID Qualifiers="0">1000</EventID>
    <Level>2</Level>
    <Task>100</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2012-03-15T17:42:42.000000000Z" />
    <EventRecordID>2945</EventRecordID>
    <Channel>Application</Channel>
    <Computer>************</Computer>
    <Security />
  </System>
  <EventData>
    <Data>Syslogd_Service.exe</Data>
    <Data>9.2.0.1</Data>
    <Data>4d069c0f</Data>
    <Data>unknown</Data>
    <Data>0.0.0.0</Data>
    <Data>00000000</Data>
    <Data>c0000005</Data>
    <Data>0000000a</Data>
    <Data>91d0</Data>
    <Data>01cd02c944ab6d53</Data>
    <Data>C:\Program Files (x86)\Syslogd\Syslogd_Service.exe</Data>
    <Data>unknown</Data>
    <Data>43e40d87-6ec6-11e1-a52f-3cd92b024752</Data>
  </EventData>
</Event>

---------------------------

The following was in the Syslogd Errorlog.txt:

2012-03-15 09:32:52    Command line license key accepted.
2012-03-15 10:42:41    *** INTERNAL PROGRAM ERROR - Please contact http://www.kiwisyslog.com/support/ ***
2012-03-15 10:42:41    Service Version 9.2.1 | Error Number: 28 | Description: Out of stack space | Module Name: Syslogdsvc.frm | Procedure Name: SyslogSocket_DataArrival | Line Number: 260 | Date and time: 3/15/2012 10:42:41 AM
---------------------------

I have opened SolarWinds case #323438 regarding this.

0 Kudos
1 Solution

mlan,

Using default settings, Kiwi Syslog Server can handle up to 2 million messages per hour.  More on this is found here:

http://knowledgebase.solarwinds.com/kb/questions/7/How+many+messages+can+Kiwi+Syslog+Daemon+handle%3...

As you are sending it close to 4 times that amount, I can only recommend balancing the load over 2 installations or more (see http://www.kiwisyslog.com/index.php?option=com_kb&page=articles&articleid=197&Itemid=244)and perhaps disabling some of the message levels.

Sincerely,

Chris Foley | Support Representative
SolarWinds | IT Management, Inspired By You
Support:866.530.8040 || Fax:512.857.0125

View solution in original post

0 Kudos
9 Replies
Level 8

Fodome,

Earlier you posted a link on balancing 2 or more installations to handle high loads.  However the link no longer works.  Is there a new link to that article?

Thanks!

0 Kudos

This might be the article:  Success Center

This really isn't balancing, it's just using more than 1 Kiwi server to handle the load.  We balance our servers using an F5(we already had it for other purposes).  You could use round-robin DNS, HAproxy or possibly Windows NLB(never used it).  The biggest issue with trying to load balance UDP is the session timeouts on the balancer.  They should be zero if possible to remove any 'stickiness' with the connection. Otherwise the load doesn't get spread well. 

0 Kudos
Level 13

mlan,

The error seems to indicate that you are sending too many messages to the Kiwi Syslog Server all at once.  Can you possibly go to "Manage -> Debug Options -> Get Diagnostics Information" and post the contents of that file here for review?

Thanks,

Chris Foley | Support Representative
SolarWinds | IT Management, Inspired By You
Support:866.530.8040 || Fax:512.857.0125

0 Kudos

Fodome,

Thanks for the reply.  I have pasted the Syslog_Diagnostics.txt below.  First off, yes, it's almost entirely Informational syslogs from two firewalls, but that is exactly what we want to capture.  At this point, I am not looking to trim down the amount of syslog traffic, but rather to find a hw/sw solution that can handle this amount of firewall traffic (~7million/hour).  Please advise if there is a recommend max traffic for Kiwi Syslog and/or SolarWinds Syslog Service.

Thanks!

 

Kiwi Syslog Server [Licensed] Version 9.2.1


///       Kiwi Syslog Server Statistics         ///
---------------------------------------------------
24 hour period ending on: Fri, 16 Mar 2012 13:35:53
Syslog Server started on: Fri, 16 Mar 2012 10:23:00
Syslog Server uptime:     3 hours, 12 minutes
---------------------------------------------------

+ Messages received - Total:          23139491
+ Messages received - Last 24 hours:  23139491
+ Messages received - Since Midnight: 23139491
+ Messages received - Last hour:      7202691
+ Message queue overflow - Last hour: 8982415
+ Messages received - This hour:      1531777
+ Message queue overflow - This hour: 1940877
+ Messages per hour - Average:        7202571

+ Messages forwarded:                 0
+ Messages logged to disk:            23139254

+ Errors - Logging to disk:           0
+ Errors - Invalid priority tag:      0
+ Errors - No priority tag:           0
+ Errors - Oversize message:          0

+ Disk space remaining on drive C:    48617 MB

---------------------------------------------------


     Breakdown of Syslog messages by sending host 
+--------------------------+------------+------------+
| Top 20 Hosts             |  Messages  | Percentage |
+--------------------------+------------+------------+
| 172.16.0.2               |  15428451  |     66.68% |
| 172.16.0.3               |   7706019  |     33.30% |
| 10.159.1.82              |       857  |      0.00% |
| 10.151.254.254           |       470  |      0.00% |
| 10.184.254.254           |       447  |      0.00% |
| 10.162.254.254           |       443  |      0.00% |
| 10.175.254.254           |       443  |      0.00% |
| 10.234.254.254           |       443  |      0.00% |
| 10.174.1.11              |       422  |      0.00% |
| 10.188.254.254           |       237  |      0.00% |
| 10.220.254.254           |       216  |      0.00% |
| 10.178.254.254           |       207  |      0.00% |
| 10.135.254.254           |       161  |      0.00% |
| 10.214.1.31              |        40  |      0.00% |
| 172.16.0.1               |        38  |      0.00% |
| 10.156.1.31              |        35  |      0.00% |
| 10.211.1.21              |        29  |      0.00% |
| 10.186.1.72              |        27  |      0.00% |
| 10.206.1.51              |        25  |      0.00% |
| 10.162.1.12              |        23  |      0.00% |
| All others (96)          |       458  |      0.00% |
+--------------------------+------------+------------+


    Breakdown of Syslog messages by severity  
+--------------------+------------+------------+
| Message Level      |  Messages  | Percentage |
+--------------------+------------+------------+
| 0 - Emerg          |         6  |      0.00% |
| 1 - Alert          |       125  |      0.00% |
| 2 - Critical       |         2  |      0.00% |
| 3 - Error          |      2170  |      0.01% |
| 4 - Warning        |    405707  |      1.75% |
| 5 - Notice         |         2  |      0.00% |
| 6 - Info           |  22347085  |     96.58% |
| 7 - Debug          |    384394  |      1.66% |
+--------------------+------------+------------+

Custom statistics
-----------------
CustomStats01: 0
CustomStats02: 0
CustomStats03: 0
CustomStats04: 0
CustomStats05: 0
CustomStats06: 0
CustomStats07: 0
CustomStats08: 0
CustomStats09: 0
CustomStats10: 0
CustomStats11: 0
CustomStats12: 0
CustomStats13: 0
CustomStats14: 0
CustomStats15: 0
CustomStats16: 0

End of Report.


DNS Cache size        20000
DNS Cache entries    0
Entries in queue    0
DNS Cache hits        0
DNS Cache misses    0
DNS Cache TTL        1440 minutes
Total DNS Lookups    0
Successful cache hits    0%




Message Buffer Information
==========================
Message Queue Max Size: 500000
Message Queue overflow: 28743810
Message Count:          499998
Message Count Max:      500000
Percentage free:        1



E-mail Buffer Information
==========================
Message Queue Max Size: 1000
Message Queue overflow: 0
Message Count:          0
Message Count Max:      0
Percentage free:        100


End of Diagnostics report

0 Kudos

I'd like to pull in these stats with SAM to be able to have historical track record and possibly alert on those - has anyone been thinking or done anything like that along the same lines? Are these counts available over WMI/SNMP somehow?

[extract from above]:

----------------------------------------------------------------------

+ Messages received - Total:          23139491

+ Messages received - Last 24 hours:  23139491

+ Messages received - Since Midnight: 23139491

+ Messages received - Last hour:      7202691

+ Message queue overflow - Last hour: 8982415

+ Messages received - This hour:      1531777

+ Message queue overflow - This hour: 1940877

+ Messages per hour - Average:        7202571

+ Messages forwarded:                 0

+ Messages logged to disk:            23139254

+ Errors - Logging to disk:           0

+ Errors - Invalid priority tag:      0

+ Errors - No priority tag:           0

+ Errors - Oversize message:          0

+ Disk space remaining on drive C:    48617 MB

----------------------------------------------------------------------

With Gratitude,

Alex Soul

0 Kudos

I'm not sure what SAM is but you can 'export' the stats via a script.  You need to get the variable 'Fields.GetDailyStatistics'.  Call the script on the schedule you need then parse the results.

0 Kudos

Parsing text to extract values ?!

pastedImage_0.png

I was hoping these stats are available over SNMP, PerfMon, WMI, Bash or alike. 

By the way, SAM is one of the SolarWind's modules for Applications Monitoring, which can extract this data... well, including that dirty parsing option as well ... but I am not going this route for sure

Server & Application Monitor | SolarWinds

0 Kudos

mlan,

Using default settings, Kiwi Syslog Server can handle up to 2 million messages per hour.  More on this is found here:

http://knowledgebase.solarwinds.com/kb/questions/7/How+many+messages+can+Kiwi+Syslog+Daemon+handle%3...

As you are sending it close to 4 times that amount, I can only recommend balancing the load over 2 installations or more (see http://www.kiwisyslog.com/index.php?option=com_kb&page=articles&articleid=197&Itemid=244)and perhaps disabling some of the message levels.

Sincerely,

Chris Foley | Support Representative
SolarWinds | IT Management, Inspired By You
Support:866.530.8040 || Fax:512.857.0125

View solution in original post

0 Kudos

Fodome,

 

Thank you for the response and the information provided.

0 Kudos