5 Replies Latest reply: Oct 9, 2013 3:00 PM by tlogsdon RSS

    Syslogd_Service.exe crash - out of stack space

    mlan

      I am evaluating Kiwi Syslogd to front-end and filter syslog traffic since we are having performance problems and service crashes using the NPM Syslog Service.  Here is the hardware platform:

      HP DL385G7
      2x AMD Opteron 6174 2.2GHz 12-core processors
      32GB memory
      RAID-1 for OS/Syslog
      Windows Server 2008 R2 x64 Enterprise SP1

      I installed Kiwi Syslogd and it ran for about an hour before it crashed with this failure:


      Log Name:      Application
      Source:        Application Error
      Date:          3/15/2012 10:42:42 AM
      Event ID:      1000
      Task Category: (100)
      Level:         Error
      Keywords:      Classic
      User:          N/A
      Computer:      *********
      Description:
      Faulting application name: Syslogd_Service.exe, version: 9.2.0.1, time stamp: 0x4d069c0f
      Faulting module name: unknown, version: 0.0.0.0, time stamp: 0x00000000
      Exception code: 0xc0000005
      Fault offset: 0x0000000a
      Faulting process id: 0x91d0
      Faulting application start time: 0x01cd02c944ab6d53
      Faulting application path: C:\Program Files (x86)\Syslogd\Syslogd_Service.exe
      Faulting module path: unknown
      Report Id: 43e40d87-6ec6-11e1-a52f-3cd92b024752
      Event Xml:
      <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
        <System>
          <Provider Name="Application Error" />
          <EventID Qualifiers="0">1000</EventID>
          <Level>2</Level>
          <Task>100</Task>
          <Keywords>0x80000000000000</Keywords>
          <TimeCreated SystemTime="2012-03-15T17:42:42.000000000Z" />
          <EventRecordID>2945</EventRecordID>
          <Channel>Application</Channel>
          <Computer>************</Computer>
          <Security />
        </System>
        <EventData>
          <Data>Syslogd_Service.exe</Data>
          <Data>9.2.0.1</Data>
          <Data>4d069c0f</Data>
          <Data>unknown</Data>
          <Data>0.0.0.0</Data>
          <Data>00000000</Data>
          <Data>c0000005</Data>
          <Data>0000000a</Data>
          <Data>91d0</Data>
          <Data>01cd02c944ab6d53</Data>
          <Data>C:\Program Files (x86)\Syslogd\Syslogd_Service.exe</Data>
          <Data>unknown</Data>
          <Data>43e40d87-6ec6-11e1-a52f-3cd92b024752</Data>
        </EventData>
      </Event>

      ---------------------------

      The following was in the Syslogd Errorlog.txt:

      2012-03-15 09:32:52    Command line license key accepted.
      2012-03-15 10:42:41    *** INTERNAL PROGRAM ERROR - Please contact http://www.kiwisyslog.com/support/ ***
      2012-03-15 10:42:41    Service Version 9.2.1 | Error Number: 28 | Description: Out of stack space | Module Name: Syslogdsvc.frm | Procedure Name: SyslogSocket_DataArrival | Line Number: 260 | Date and time: 3/15/2012 10:42:41 AM
      ---------------------------

      I have opened SolarWinds case #323438 regarding this.

        • Re: Syslogd_Service.exe crash - out of stack space
          Fodome

          mlan,

          The error seems to indicate that you are sending too many messages to the Kiwi Syslog Server all at once.  Can you possibly go to "Manage -> Debug Options -> Get Diagnostics Information" and post the contents of that file here for review?

          Thanks,

          Chris Foley | Support Representative
          SolarWinds | IT Management, Inspired By You
          Support:866.530.8040 || Fax:512.857.0125

            • Re: Syslogd_Service.exe crash - out of stack space
              mlan

              Fodome,

              Thanks for the reply.  I have pasted the Syslog_Diagnostics.txt below.  First off, yes, it's almost entirely Informational syslogs from two firewalls, but that is exactly what we want to capture.  At this point, I am not looking to trim down the amount of syslog traffic, but rather to find a hw/sw solution that can handle this amount of firewall traffic (~7million/hour).  Please advise if there is a recommend max traffic for Kiwi Syslog and/or SolarWinds Syslog Service.

              Thanks!

               

              Kiwi Syslog Server [Licensed] Version 9.2.1


              ///       Kiwi Syslog Server Statistics         ///
              ---------------------------------------------------
              24 hour period ending on: Fri, 16 Mar 2012 13:35:53
              Syslog Server started on: Fri, 16 Mar 2012 10:23:00
              Syslog Server uptime:     3 hours, 12 minutes
              ---------------------------------------------------

              + Messages received - Total:          23139491
              + Messages received - Last 24 hours:  23139491
              + Messages received - Since Midnight: 23139491
              + Messages received - Last hour:      7202691
              + Message queue overflow - Last hour: 8982415
              + Messages received - This hour:      1531777
              + Message queue overflow - This hour: 1940877
              + Messages per hour - Average:        7202571

              + Messages forwarded:                 0
              + Messages logged to disk:            23139254

              + Errors - Logging to disk:           0
              + Errors - Invalid priority tag:      0
              + Errors - No priority tag:           0
              + Errors - Oversize message:          0

              + Disk space remaining on drive C:    48617 MB

              ---------------------------------------------------


                   Breakdown of Syslog messages by sending host 
              +--------------------------+------------+------------+
              | Top 20 Hosts             |  Messages  | Percentage |
              +--------------------------+------------+------------+
              | 172.16.0.2               |  15428451  |     66.68% |
              | 172.16.0.3               |   7706019  |     33.30% |
              | 10.159.1.82              |       857  |      0.00% |
              | 10.151.254.254           |       470  |      0.00% |
              | 10.184.254.254           |       447  |      0.00% |
              | 10.162.254.254           |       443  |      0.00% |
              | 10.175.254.254           |       443  |      0.00% |
              | 10.234.254.254           |       443  |      0.00% |
              | 10.174.1.11              |       422  |      0.00% |
              | 10.188.254.254           |       237  |      0.00% |
              | 10.220.254.254           |       216  |      0.00% |
              | 10.178.254.254           |       207  |      0.00% |
              | 10.135.254.254           |       161  |      0.00% |
              | 10.214.1.31              |        40  |      0.00% |
              | 172.16.0.1               |        38  |      0.00% |
              | 10.156.1.31              |        35  |      0.00% |
              | 10.211.1.21              |        29  |      0.00% |
              | 10.186.1.72              |        27  |      0.00% |
              | 10.206.1.51              |        25  |      0.00% |
              | 10.162.1.12              |        23  |      0.00% |
              | All others (96)          |       458  |      0.00% |
              +--------------------------+------------+------------+


                  Breakdown of Syslog messages by severity  
              +--------------------+------------+------------+
              | Message Level      |  Messages  | Percentage |
              +--------------------+------------+------------+
              | 0 - Emerg          |         6  |      0.00% |
              | 1 - Alert          |       125  |      0.00% |
              | 2 - Critical       |         2  |      0.00% |
              | 3 - Error          |      2170  |      0.01% |
              | 4 - Warning        |    405707  |      1.75% |
              | 5 - Notice         |         2  |      0.00% |
              | 6 - Info           |  22347085  |     96.58% |
              | 7 - Debug          |    384394  |      1.66% |
              +--------------------+------------+------------+

              Custom statistics
              -----------------
              CustomStats01: 0
              CustomStats02: 0
              CustomStats03: 0
              CustomStats04: 0
              CustomStats05: 0
              CustomStats06: 0
              CustomStats07: 0
              CustomStats08: 0
              CustomStats09: 0
              CustomStats10: 0
              CustomStats11: 0
              CustomStats12: 0
              CustomStats13: 0
              CustomStats14: 0
              CustomStats15: 0
              CustomStats16: 0

              End of Report.


              DNS Cache size        20000
              DNS Cache entries    0
              Entries in queue    0
              DNS Cache hits        0
              DNS Cache misses    0
              DNS Cache TTL        1440 minutes
              Total DNS Lookups    0
              Successful cache hits    0%




              Message Buffer Information
              ==========================
              Message Queue Max Size: 500000
              Message Queue overflow: 28743810
              Message Count:          499998
              Message Count Max:      500000
              Percentage free:        1



              E-mail Buffer Information
              ==========================
              Message Queue Max Size: 1000
              Message Queue overflow: 0
              Message Count:          0
              Message Count Max:      0
              Percentage free:        100


              End of Diagnostics report

            • Re: Syslogd_Service.exe crash - out of stack space
              tlogsdon

              Fodome,

              Earlier you posted a link on balancing 2 or more installations to handle high loads.  However the link no longer works.  Is there a new link to that article?

               

              Thanks!