14 Replies Latest reply on Feb 16, 2017 3:40 PM by swf5002@psu.edu

    Last Database Update is in the past, Next Poll... More past'er'er

    zhollett

      So, as you can see, the next poll is 2 days in the past from the last database update.

      Normal?  I dont remember the system reaching 88 mph.

      Volume Polling Details
        Polling Interval 120 seconds 
        Next Poll 06-Aug-2011 01:47 PM 
       
        Statistics Collection 15 minutes 
       
        Rediscovery Interval 30 minutes 
        Next Rediscovery 06-Aug-2011 02:00 PM 
       
        Last Database Update 08-Aug-2011 09:26 AM 

        • Re: Last Database Update is in the past, Next Poll... More past'er'er
          mdriskell

          This isn't normal and I would suspect your flux capacitor is misaligned ;)

          I had these issues quite often due to polling to frequently for the size of our network.  You could open a ticket with support to see why this is happening.

          Now what I would always do to get things back on track was to follow the process laid on on this post SolarWinds Collector Data Processor/Polling Controller restarts all the time

          If it continues to happen you may have other underlying problems to deal with.  I created a report to show me the status of a last device poll on our main page so I could quickly start to detect a lag...my issues were due to an undersized SQL box not being able to keep up with the load.

          1 of 1 people found this helpful
            • Re: Last Database Update is in the past, Next Poll... More past'er'er
              zhollett

              This is what the front page of my Home screen says about the polling.  Seems pretty up to date, the system isn't lagging on the polling so I dont think thats an issue.  The (virtual) SQL box has 8gb/dual 2.7ghz's. I wonder if its because it's a vm instead of physical.

              Polling Engine Status

              Last Database Update Now
              Polling Engine on   Main Orion Server
              IP Address  {super.secret.ip.address} 
              Last Database Sync  2 seconds ago 
              Network Elements  602 Nodes, 595 Interfaces, 1587 Volumes, 2784 Total Elements 
              Polling Completion  99.80% 
              Operating System  
              Service Pack  0.0 
              Package  
              Polling Engine on   Polling Engine 1
              IP Address  {super.secret.ip.address} 
              Last Database Sync  3 seconds ago 
              Network Elements  688 Nodes, 1128 Interfaces, 1455 Volumes, 3271 Total Elements 
              Polling Completion  99.70% 
              Operating System  Windows Server 2008 R2 
              Service Pack  
              Package  
              Polling Engine on   Polling Engine 2
              IP Address  {super.secret.ip.address} 
              Last Database Sync  2 seconds ago 
              Network Elements  773 Nodes, 1490 Interfaces, 2777 Volumes, 5040 Total Elements 
              Polling Completion  99.76% 
              Operating System  Windows Server 2008 R2 
              Service Pack  
              Package 

            • Re: Last Database Update is in the past, Next Poll... More past'er'er
              MNPS_G

              Is that the case for nodes on every poller, or a specific poller?

              If it is a specific poller, have you checked the messaging queues on that poller? 

              I believe I had that exactly problem on a poller that was overloaded - it was queuing up the polling data and was writing data to the SQL server slower than it was being queued up, causing a backlog.  The nodes on that server had next polling times in the past.

              Geoffrey

                • Re: Last Database Update is in the past, Next Poll... More past'er'er
                  mdriskell

                  zhollett,

                  a few things stand out to me about your configuration.  You have three polling engines polling approximately 1200 nodes and 11000 elements.  Your SQL server could very well be your bottleneck if it only has 8GB of RAM.  Have you looked into your SQL performance?

                  MNPS's comment about checking the message queuing could be a very good place to look.  The polling engine could very well be able to poll the nodes quick enough but waiting on the SQL server to write to the database.  If you check the message queues you may see writes to the database from two days ago.  In those instances the pollers are keeping up but you are looking at data in the past because the updates are still sitting in queue waiting to be written.

                  The final concern I have is your rediscovery timer is 30 minutes....I believe the factory default is 360 minutes so that would be a very aggressive 12 times more often then Solarwinds recommends.

                  Opening a ticket with support would probably be the quickest way to isolate but with what you have posted IMHO these would be your usual suspects.

                    • Re: Last Database Update is in the past, Next Poll... More past'er'er
                      zhollett

                      Well, I opened a support ticket and Radu was very helpful in resolving the "polling dates in the past" issue. Here is what was accomplished.

                      REPLACE JOB ENGINE
                      1. Stop all Orion services from Orion Service Manager.
                      2. Go to C:\Documents and Settings\All Users\Application Data\SolarWinds\JobEngineV2\Data\JobEngine35.sdf file and rename it to JobEngine35_OLD.sdf
                      3. Then create a copy of C:\Documents and Settings\All Users\Application Data\SolarWinds\JobEngineV2\Data\JobEngine35 - Blank.sdf one and rename it to JobEngine35.sdf.
                      4. This way you have the old one to revert back to and you always have a blank copy in case issue reoccurs.
                      5. Right click on properties of the JobEngine35.sdf you have created and untick the read only box and click ok.

                      REPLACE JOB TRACKER
                      1. Stop all Orion services from Orion Service Manager.
                      2. Go to C:\Documents and Settings\All Users\Application Data\SolarWinds\Collector\Data\JobTracker.sdf file and rename it to JobTracker_OLD.sdf
                      3. Then create a copy of C:\Documents and Settings\All Users\Application Data\SolarWinds\Collector\Data\JobTracker - Blank.sdf one and rename it to JobTracker.sdf.
                      4. This way you have the old one to revert back to and you always have a blank copy in case issue reoccurs.
                      5. Right click on properties of the JobTracker.sdf you have created and untick the read only box and click ok.
                       
                      REPLACE POLLING COLLECTOR
                      1. Stop all Orion services from Orion Service Manager.
                      2. Go to C:\Documents and Settings\All Users\Application Data\SolarWinds\Collector\Data\PollingController.sdf file and rename it to PollingController_OLD.sdf
                      3. Then create a copy of C:\Documents and Settings\All Users\Application Data\SolarWinds\Collector\Data\PollingController - Blank.sdf one and rename it to PollingController.sdf.
                      4. This way you have the old one to revert back to and you always have a blank copy in case issue reoccurs.
                      5. Right click on properties of the PollingController.sdf you have created and untick the read only box and click ok.
                      6. Restart all Orion Services

                      Hope this helps anyone running into this issue.

                      (now, where did I put that flux-capacitor...)

                      1 of 1 people found this helpful
                  • Re: Last Database Update is in the past, Next Poll... More past'er'er
                    mwb

                    Anyone know where JobTracker.sdf has shifted to?  It no longer appears to reside with the PollingController.sdf.

                    • Re: Last Database Update is in the past, Next Poll... More past'er'er
                      swf5002@psu.edu

                      Hey just FYI, don't do _OLD.sdf as the new name, you have to change the extension to .old.  I guess solarwinds is just looking for an .sdf file type rather than the name itself.  Learned that from calling support on a different issue.