Connection Timeout. Job Canceled by Scheduler. APM 2.5

Has anyone successfully been able to correct this on APM 2.5? We have approx 600 components with an unlimited license, and I get 4000 alearts per day with this. The monitors wil come up, then down. My timeouts are at an absurd 20 minutes.

We've opened a case with SW support, but our feeling is they don't really know what's going on and don't seem too committed to helping us resolve the issue (Case: 90187).

Yes, we've run all local and remote WMI tests and everything works great there. We are trying to monitor applications on servers on the same switch, without success.

We've replaced the polling engine

Rebooted Servers

Replaced the SQL Server file

Any ideas?

Find more posts tagged with

timeout

apm_2.5

Accepted answers

All comments

kasaff

Hello Ken,

I'm the support engineer that has been working on this case with Craig. As I've communicated to him via the case, we are committed to working with you to resolve this. I understand your frustration with the amount of time it is taking, but unfortunately it is not a simple fix or an easy issue to pinpoint. Our APM development team is working on a resolution and I will let you know as soon I have more information for you.

FormerMember

Has anyone successfully been able to correct this on APM 2.5? We have approx 600 components with an unlimited license, and I get 4000 alearts per day with this. The monitors wil come up, then down. My timeouts are at an absurd 20 minutes.

Based on our Support cases, very few users are experiencing this problem. If anyone else is experiencing it, please open a Support ticket. We need more data.

We've opened a case with SW support, but our feeling is they don't really know what's going on and don't seem too committed to helping us resolve the issue (Case: 90187).

You are half right. We do not know the cause. If we did, we would have issued a fix. We want to work with each of the several cusotmers who have the issue. Because it's intermittent, we need a lot more data, so we need to do some heavy logging. It is likely that it has something to do with WMI.

You are incorrect to say that we are not very committed to resolving the issue. We will fix the issue. To be perfectly honest, it's just a hard issue to troubleshoot. Once we know the problem, the fix is usually quick, but if we can't reproduce it (and we can't), it takes a lot longer to diagnose.

kenplunkett

Well... I'm seriously dying with this. I'd expect if this was important or we were truly focused on getting the issue resolved, I'd get more than 1 update every 3-4 days (if that). Usually, we have to hound someone down to get that.

I'm not sure what "intermittent" means, but as I mentioned earlier I get 4000 alerts PER DAY. I seriously doubt it's an issue with WMI - probably more an issue with how you're working with WMI. We have another software tool that I've since loaded on servers were APM had issues and they work just fine.

If you need logging or more data, we're certainly happy to help, but for the last 3 weeks all I've been asked for is to run through WMI testing, which works fine.

My frustration level is high because I'm accountable for this, and I really don't have an update that does any good. I don't know whether to ride this out, or ask for a refund and go with another vendor.

HELP!

klomper2007

Ken,

I'm one of the other cases that they are working with to address this issue. I can tell you (from an enduser standpoint) that they are indeed working on the issue. I agree that it'd be nice if they were able to come up with a quicker resolution, but as is life in the IT world :-)

I did want to give you an idea that is working for us at this point, until they are able to fix the issue. In our alerts for applications we are only alerting when the "Component Status is equal to Down". So we don't get alerts for the unknown status. It's a thought...maybe you have things setup different then we do. Obviously you will still see thing when they go into the "unknown" status in the events, but at least you don't get alerted.

Thanks.

FormerMember

Ken,

Klomper2007 is right about adjusting the alerts so that you're not getting alerted on "unknown". If you don't want to do that, another way to suppress the alerts is adjust the interval on the Trigger Condition tab so that it won't alert until a condition has been true for 10 minutes (which I assume is 2 polling intervals), which would stop alerts from transient "unknowns", but which would lengthen time to alert for true alerts. I like Klomper2007's solution, but there are options.

The engineering team is going to try to see if there's something else that they can do short-term to alleviate the problem (or, better yet, fix it if we can pin it down).

kenplunkett

Any additional news... I ran the debugs and tests that were requested of us on 5.14.09. The only updates I get are "no updates" and only after I ask for a follow up. This isn't consistant with an organization comitted to helping.

Please advise... What's going on???

thunter99

My organization is also hoping this issue is resolved. We see 20-30% of the WMI statistic monitors being canceled every other polling cycle.

We added a second poller, fixed some database performance issues and put gigabit NICs in the orion boxes but none of these helped this issue.

Although you can suppress the alerts, the dashboard icons still show grey for unknowns which confuses people and my reports have a lot of data gaps.

I've been running Sitescope and Zenoss remote WMI performance monitors across the same environment with no problems.

FormerMember

This is also becoming quite serious for us as well. We also have had a ticket open (90802) since 4/15. We need some kind of time frame as to when this issue will be resolved. I am in the process of purchashing 2 more Orion servers (SLX) with APM and EOC and I will need to hold up that purchase until I know that this isuse is resolved. When I call in I am told "We are working on it".

Throw me a bone...Please

ElisabethZakes

I wish we could give definite timelines on things like this, but if a case has been escalated to development, all we can say is, "It'll be done when they finish." We will certainly update any cases still open that are attached to the issue. They ARE working on it, that I know. But that's all we can say until they tell us that it's been fixed. You're certainly free to call or e-mail your technician for an update, though!

kenplunkett

Elizabeth - I'm really sorry, but "we're working on it" just isn't acceptable. That works for a week or two, but this has now been about 2 months for us. I need some timelines and some commitments, and I think I'm entitled to that. Remember, you have my money and in return provided me a product that doesn't work. Please provide me something more concrete.

FormerMember

Ken,

This issue has been extremely hard to reproduce in house. Without the ability to reproduce at will, we have to work on various hypotheses about what's going on. We created what we thought was a fix, and tried it with a couple of cusotmers, but it didn't fix the problem. We are now re-examining diagnostics and working on other hypothesis. That's as concrete as we can get.

So we are, in fact, committed to solving the problem. Unfortunately, we can't go any faster that we're already going. I appreciate that you're frustrated, and in your shoes, I would be too. We're frustrated that we can't find the bug. We will keep working on it, and working with you.

thunter99

It looks like this problem has been solved in my environment. While working on an unrelated ticket (#106550) the support engineer noticed a large SW job engine file size (i.e. 92 MB)...I can't remember the exact file name. He reinstalled it and my issues with skipped WMI polling and HTML application template errors disappeared.

robgarbee

OK, So we have started to see this issue as well. I'm on hold with support now. Has anyone heard anything? Judging by the post here it doesn't look to good.

-Rob

robgarbee

Hey support, please help!!!! My boss is talking crazy because of this...He's saying things like NAGIOS! Hurry please before it's to late.

FormerMember

Roger,

If you look in your portal, you will find Release Candidate 1 for APM 3.0. The issue should be fixed in there. Support will take cases on RC-1. We expect that RC-1 will be the same as GA, but if we find any issues, we reserve the right to make changes. We will support upgrades from RC-1 to GA.

I'm recommend you try that.

TGhosh

I've recently upgraded to APM 3.0 RC from APM 2.5. Its still too early to call it a victory, but my IIS alerts due to Connection Timeout have stopped. General look and feel of APM is much better as well. I also upgraded NPM to 9.5 from 9.1 and I'm happy so far.

aLTeReGo

AMP 3.x has been out for a while now. Did that resolve your problem? Is this still an open issue?