5 Replies Latest reply: Oct 9, 2013 3:00 PM by tlogsdon RSS

Syslogd_Service.exe crash - out of stack space

mlan

I am evaluating Kiwi Syslogd to front-end and filter syslog traffic since we are having performance problems and service crashes using the NPM Syslog Service.  Here is the hardware platform:

HP DL385G7
2x AMD Opteron 6174 2.2GHz 12-core processors
32GB memory
RAID-1 for OS/Syslog
Windows Server 2008 R2 x64 Enterprise SP1

I installed Kiwi Syslogd and it ran for about an hour before it crashed with this failure:


Log Name:      Application
Source:        Application Error
Date:          3/15/2012 10:42:42 AM
Event ID:      1000
Task Category: (100)
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      *********
Description:
Faulting application name: Syslogd_Service.exe, version: 9.2.0.1, time stamp: 0x4d069c0f
Faulting module name: unknown, version: 0.0.0.0, time stamp: 0x00000000
Exception code: 0xc0000005
Fault offset: 0x0000000a
Faulting process id: 0x91d0
Faulting application start time: 0x01cd02c944ab6d53
Faulting application path: C:\Program Files (x86)\Syslogd\Syslogd_Service.exe
Faulting module path: unknown
Report Id: 43e40d87-6ec6-11e1-a52f-3cd92b024752
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Application Error" />
    <EventID Qualifiers="0">1000</EventID>
    <Level>2</Level>
    <Task>100</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2012-03-15T17:42:42.000000000Z" />
    <EventRecordID>2945</EventRecordID>
    <Channel>Application</Channel>
    <Computer>************</Computer>
    <Security />
  </System>
  <EventData>
    <Data>Syslogd_Service.exe</Data>
    <Data>9.2.0.1</Data>
    <Data>4d069c0f</Data>
    <Data>unknown</Data>
    <Data>0.0.0.0</Data>
    <Data>00000000</Data>
    <Data>c0000005</Data>
    <Data>0000000a</Data>
    <Data>91d0</Data>
    <Data>01cd02c944ab6d53</Data>
    <Data>C:\Program Files (x86)\Syslogd\Syslogd_Service.exe</Data>
    <Data>unknown</Data>
    <Data>43e40d87-6ec6-11e1-a52f-3cd92b024752</Data>
  </EventData>
</Event>

---------------------------

The following was in the Syslogd Errorlog.txt:

2012-03-15 09:32:52    Command line license key accepted.
2012-03-15 10:42:41    *** INTERNAL PROGRAM ERROR - Please contact http://www.kiwisyslog.com/support/ ***
2012-03-15 10:42:41    Service Version 9.2.1 | Error Number: 28 | Description: Out of stack space | Module Name: Syslogdsvc.frm | Procedure Name: SyslogSocket_DataArrival | Line Number: 260 | Date and time: 3/15/2012 10:42:41 AM
---------------------------

I have opened SolarWinds case #323438 regarding this.

 
  • Re: Syslogd_Service.exe crash - out of stack space
    Fodome

    mlan,

    The error seems to indicate that you are sending too many messages to the Kiwi Syslog Server all at once.  Can you possibly go to "Manage -> Debug Options -> Get Diagnostics Information" and post the contents of that file here for review?

    Thanks,

    Chris Foley | Support Representative
    SolarWinds | IT Management, Inspired By You
    Support:866.530.8040 || Fax:512.857.0125

    • Re: Syslogd_Service.exe crash - out of stack space
      mlan

      Fodome,

      Thanks for the reply.  I have pasted the Syslog_Diagnostics.txt below.  First off, yes, it's almost entirely Informational syslogs from two firewalls, but that is exactly what we want to capture.  At this point, I am not looking to trim down the amount of syslog traffic, but rather to find a hw/sw solution that can handle this amount of firewall traffic (~7million/hour).  Please advise if there is a recommend max traffic for Kiwi Syslog and/or SolarWinds Syslog Service.

      Thanks!

       

      Kiwi Syslog Server [Licensed] Version 9.2.1


      ///       Kiwi Syslog Server Statistics         ///
      ---------------------------------------------------
      24 hour period ending on: Fri, 16 Mar 2012 13:35:53
      Syslog Server started on: Fri, 16 Mar 2012 10:23:00
      Syslog Server uptime:     3 hours, 12 minutes
      ---------------------------------------------------

      + Messages received - Total:          23139491
      + Messages received - Last 24 hours:  23139491
      + Messages received - Since Midnight: 23139491
      + Messages received - Last hour:      7202691
      + Message queue overflow - Last hour: 8982415
      + Messages received - This hour:      1531777
      + Message queue overflow - This hour: 1940877
      + Messages per hour - Average:        7202571

      + Messages forwarded:                 0
      + Messages logged to disk:            23139254

      + Errors - Logging to disk:           0
      + Errors - Invalid priority tag:      0
      + Errors - No priority tag:           0
      + Errors - Oversize message:          0

      + Disk space remaining on drive C:    48617 MB

      ---------------------------------------------------


           Breakdown of Syslog messages by sending host 
      +--------------------------+------------+------------+
      | Top 20 Hosts             |  Messages  | Percentage |
      +--------------------------+------------+------------+
      | 172.16.0.2               |  15428451  |     66.68% |
      | 172.16.0.3               |   7706019  |     33.30% |
      | 10.159.1.82              |       857  |      0.00% |
      | 10.151.254.254           |       470  |      0.00% |
      | 10.184.254.254           |       447  |      0.00% |
      | 10.162.254.254           |       443  |      0.00% |
      | 10.175.254.254           |       443  |      0.00% |
      | 10.234.254.254           |       443  |      0.00% |
      | 10.174.1.11              |       422  |      0.00% |
      | 10.188.254.254           |       237  |      0.00% |
      | 10.220.254.254           |       216  |      0.00% |
      | 10.178.254.254           |       207  |      0.00% |
      | 10.135.254.254           |       161  |      0.00% |
      | 10.214.1.31              |        40  |      0.00% |
      | 172.16.0.1               |        38  |      0.00% |
      | 10.156.1.31              |        35  |      0.00% |
      | 10.211.1.21              |        29  |      0.00% |
      | 10.186.1.72              |        27  |      0.00% |
      | 10.206.1.51              |        25  |      0.00% |
      | 10.162.1.12              |        23  |      0.00% |
      | All others (96)          |       458  |      0.00% |
      +--------------------------+------------+------------+


          Breakdown of Syslog messages by severity  
      +--------------------+------------+------------+
      | Message Level      |  Messages  | Percentage |
      +--------------------+------------+------------+
      | 0 - Emerg          |         6  |      0.00% |
      | 1 - Alert          |       125  |      0.00% |
      | 2 - Critical       |         2  |      0.00% |
      | 3 - Error          |      2170  |      0.01% |
      | 4 - Warning        |    405707  |      1.75% |
      | 5 - Notice         |         2  |      0.00% |
      | 6 - Info           |  22347085  |     96.58% |
      | 7 - Debug          |    384394  |      1.66% |
      +--------------------+------------+------------+

      Custom statistics
      -----------------
      CustomStats01: 0
      CustomStats02: 0
      CustomStats03: 0
      CustomStats04: 0
      CustomStats05: 0
      CustomStats06: 0
      CustomStats07: 0
      CustomStats08: 0
      CustomStats09: 0
      CustomStats10: 0
      CustomStats11: 0
      CustomStats12: 0
      CustomStats13: 0
      CustomStats14: 0
      CustomStats15: 0
      CustomStats16: 0

      End of Report.


      DNS Cache size        20000
      DNS Cache entries    0
      Entries in queue    0
      DNS Cache hits        0
      DNS Cache misses    0
      DNS Cache TTL        1440 minutes
      Total DNS Lookups    0
      Successful cache hits    0%




      Message Buffer Information
      ==========================
      Message Queue Max Size: 500000
      Message Queue overflow: 28743810
      Message Count:          499998
      Message Count Max:      500000
      Percentage free:        1



      E-mail Buffer Information
      ==========================
      Message Queue Max Size: 1000
      Message Queue overflow: 0
      Message Count:          0
      Message Count Max:      0
      Percentage free:        100


      End of Diagnostics report

  • Re: Syslogd_Service.exe crash - out of stack space
    tlogsdon

    Fodome,

    Earlier you posted a link on balancing 2 or more installations to handle high loads.  However the link no longer works.  Is there a new link to that article?

     

    Thanks!