22 Replies Latest reply on Dec 29, 2014 9:26 AM by dsbalcau

    data limit exceeded

    dwcasey

      Seeing a lot of these on my VMWare servers. Our VCenter was rebuilt. I updated user/pass info as well as URL with IP of VCenter, but still seeing a lot of these:

       

      22 Feb 2013 16:34:17 - Data limit exceeded, removing oldest entry with key ' N ' from performance (deleted: true)

      22 Feb 2013 16:34:18 - Data limit exceeded, removing oldest entry with key ' N!' from performance (deleted: true)

      22 Feb 2013 16:34:18 - Data limit '536870912' reached for mod.sys.virtual.Virtual_1.0, sending notification to server

      22 Feb 2013 16:34:18 - Data ready for mod.sys.virtual.Virtual_1.0, sending notification to server

      22 Feb 2013 16:34:19 - Data limit exceeded, removing oldest entry with key ' N"' from performance (deleted: true)

      22 Feb 2013 16:34:20 - Data limit exceeded, removing oldest entry with key ' N#' from performance (deleted: true)

       

      SW support recommended we run an ISAMcheck and it ran and ran and ran. We've cleared out data on the agent servers and update agent config to have 2048M memory ( -xmx option ). Not sure what else to do.

        • Re: data limit exceeded
          dwcasey

          Also seeing this on the dashboard "Data Collection Issue : Connection Error.Unable to login."

            • Re: data limit exceeded
              dsbalcau

              Hi dwcasey,

               

              Can you please let me know the size of the virtual environment you are trying to monitor? # of esx hosts, # of vm's?

               

              Also, from your description, it sounds as if you've deployed at least one agent for monitoring VM's. Are there more than one agent?

                • Re: data limit exceeded
                  dwcasey

                  Sorry about formatting. we have seven agent servers and have tried to distrubute ~200 per agent server

                   

                  server | esxcount | #vms | #vmsrunning  

                  sws1 | 12 | 120 | 111

                  sws2 | 13 | 198 | 183

                  sws3 | 6 | 170 | 164

                  sws4 | 6 | 191 | 175

                  sws5 | 11 | 204 | 168

                  sws6 | 3 | 71 | 70

                  sws7 | 2 | 107 | 92

                    • Re: data limit exceeded
                      dsbalcau

                      I'm guessing that your individual agents monitoring ESX Host Resources haven't been updated with the new credentials. You can create a new VMware policy with the new creds and push the policy out to the monitoring agents or change the credentials individually. Policies can be configured from Settings and then Device Management->Policies.

                        • Re: data limit exceeded
                          dwcasey

                          Whenever I try to save existing or new VMWare/Virtualization policy, I get the following error:

                           

                          This page contains the following errors:

                          error on line 2 at column 6: XML declaration allowed only at the start of the document

                          Below is a rendering of the page up to the first error.

                            • Re: data limit exceeded
                              dsbalcau

                              Hmm, OK. You should just be able to unassign/re-assign your ESX Hosts/Clusters from the Storage Manager agents and it should reapply the proper credentials.

                               

                              Go to Settings -> Manage Agent Assignments -> Cluster/ESX in the Storage Manager Web Console.

                              1 of 1 people found this helpful
                                • Re: data limit exceeded
                                  dwcasey

                                  Unassigned and reassigned. Went to dashboard, saw I had data collection errors, went to one of the hosts with an error, "Overall status: Data Collection Issue : Connection Error. Unable to retrieve service content ". Went to Status Report, Virtualization is online, clicked on it, and have the following:

                                   

                                  26 Feb 2013 10:30:36 - VMClient : Fault Reason = The object has already been deleted or has not been completely created

                                  26 Feb 2013 10:30:36 - VMClient : Fault String = The object has already been deleted or has not been completely created

                                  26 Feb 2013 10:30:36 - VMwareAnalyzer(801 asset) :.run(): No Virtual Machines found for ESX host xx-xxx-xxxx

                                  26 Feb 2013 10:30:36 - VMClient : Managed Object is not found in the inventory in getObjectOfMOR.

                                  26 Feb 2013 10:30:36 - VMClient : Managed Object Name = host-31472

                                  26 Feb 2013 10:30:36 - VMClient : Fault Reason = The object has already been deleted or has not been completely created

                                  26 Feb 2013 10:30:36 - VMClient : Fault String = The object has already been deleted or has not been completely created

                                  26 Feb 2013 10:30:46 - Data limit exceeded, removing oldest entry with key ' \' from strg (deleted: true)

                                  26 Feb 2013 10:30:46 - Data limit '536870912' reached for mod.sys.virtual.Virtual_1.0, sending notification to server

                                  26 Feb 2013 10:30:46 - Data ready for mod.sys.virtual.Virtual_1.0, sending notification to server

                                  .

                                  .

                                  26 Feb 2013 10:31:28 - Data limit exceeded, removing oldest entry with key ' a' from strg (deleted: true)

                                  26 Feb 2013 10:31:28 - VMwareAnalyzer(7274 strg) :.run(): Total time taken for Device - 7274 is 16 seconds

                                  26 Feb 2013 10:31:29 - Data limit exceeded, removing oldest entry with key '   ' from asset (deleted: true)

                                  26 Feb 2013 10:31:29 - Data limit exceeded, removing oldest entry with key '   ' from asset (deleted: true)

                                  26 Feb 2013 10:31:29 - VMwareAnalyzer(7275 asset) :.run(): Total time taken for Device - 7275 is 15 seconds

                      • Re: data limit exceeded
                        dwcasey

                        After doing the unassigned and reassign, I decided to do what I've seen support do many times and that stop the agent service on the agent servers, clean out the 'data' directories under systemic and then start the service back up. Waiting to see if that has helped.

                         

                        Question: is it a problem that I now have two Virtualization Policies? Should I remove the new one, just leave it, doesn't matter??

                         

                        Default VMware PolicyVirtualizationEdit PolicyPush Policy
                        New VMWare PolicyVirtualizationEdit PolicyDelete PolicyPush Policy
                          • Re: data limit exceeded
                            dsbalcau

                            Well, it looks like your an old pro

                             

                            It should not matter that you have two policies. You're agents should use whatever policy was last pushed to them.

                              • Re: data limit exceeded
                                dwcasey

                                I guess, I'm just ready for these issues were fixed. So far, my event monitor does not indicate data collection issues for the last 13 minutes or so. Seeing some 'Threshold Condition Met' and that is all.

                                 

                                Is it possibly these could be overrunning the system? Too many 'alerts'? I noticed their Source is 'tt-rules-traps' and when I was doing the ISAM checks, the largest files were tt-* files...

                            • Re: data limit exceeded
                              dwcasey

                              Aaaaannd it's back... went about 20-minutes after the restarts and cleanup/restart of the agent servers and just got this a minute ago.

                               

                              Detailed Description

                              Data generation for the indicated archive has exceeded its assigned window for data. This is an indication that the agent has not been collected from for an extended period or data is being generated too quickly.

                               

                              Original Trap Source

                              TRAP DETAILS: Enterprise: 1.3.6.1.4.1.10089.5 Agent Address: xx.xx.xx.xxx Generic Trap: 6 Specific Trap: 34

                               

                              VarBindings

                              moduleName = mod.adm.control.control_1.0

                              limitSize = 10485760

                               

                              Suggested Corrective Action

                              Please verify that the agent is active and licensed for collection.

                                • Re: data limit exceeded
                                  dsbalcau

                                  OK, Support should be reaching out to you shortly.

                                   

                                  I don't think this is a matter of the Server getting overwhelmed with alerts. Basically, if the agents are collecting data faster than they can get it sent back to the Server, we will exceed the allotted "data window" that is configured on the agent and begin overwriting data in a circular buffer.

                                   

                                  It does look like the agents are collecting data, so it appears that your credentials issue is fixed. Support should be able to help resolve the other issue you are now seeing.

                                  1 of 1 people found this helpful
                                • Re: data limit exceeded
                                  dwcasey

                                  Ticket updated with information I provided here in this forum, but I have not heard back from support since last Friday.

                                    • Re: data limit exceeded
                                      dsbalcau

                                      Hi dwcasey,

                                       

                                      Support has tried reaching out to you (phone-voicemail/email) yesterday and today without success. I'm guessing that maybe our emails are getting stuck in a SPAM folder/email filter perhaps? Let me know if this continues to be a problem.

                                    • Re: data limit exceeded
                                      dwcasey

                                      I've been in touch with support and we made some progress, but the issue returned soon after they attempted corrective measures. Still waiting on escalation to developers/higher level support.

                                      • Re: data limit exceeded
                                        bharris

                                        Did you ever resolve this issue?  I'm getting the error " Unable to retrieve service content" when I drill down into one of my vCenters.

                                         

                                        EDIT: I just realized this was a year old.  Whoops...