Is it possible to have a perfect storm of issues?

lcsw2013

So I have several pain points and I was curious to know if anyone has seen worst? Or if anyone has tips, tricks, advice, help, etc..that will be well appreciated as well!

1. The case of the orphaned phantom service. 4 of our additional pollers decided that it no longer wanted to consider one of the business layer processes valid. The process does run and is started on every reboot. Force kill doesn't work, says PID is invalid. Through task manager says access denied. And it's evident that it's running as it's locking files. We know this because on these 4 pollers, configuration wizard will not run. And permissions are correct as confirmed by permissions checker and verified manually. Head scratcher isn't?

2. Primary poller randomly opening 12k plus WMI connections to whois.iana.org. weird, why do it through wmi? why is whois needed? Why not reach local dns, etc. Lots of why's. And no answers.

3. Agent communication randomly dropping out. A poller will show a server up. Show data, but it's obsolete. Agent says connected and is green. Agent restart command works. still green. Yet list resources hangs forever. And poll no won't return new data. Poller reboot all of a sudden syncs the agents back on and we are flooded with servers showing reboot alerts and properly showing data.

4. Site rendering some weirdness. On the login page we see a space at the top of the page and randomly that space will show a button with something regarding widget setup???? Ummmm, I'm confused. Then different sections of the site will randomly show errors. Configuration wizard is clean on primary and site has been rebuilt at least 30 times already with the same effect.

5. When database maintenance runs alot of connection timeouts to the database during the process. Can see the server, can ping the server, just can't continue to sql saying connection timed out waiting response. Because of this we have frequent pollers failures among issues with data consistency and accuracy. DB was built with all solarwinds recommendations and than some more. Has a 10gig network pipe and fastest storage we have available in our infrastructure.

6. IPAM is a complete mess. Unusable.

7. Alerts work weirdly. You clear one alert the system automatically clears every thing after it and retriggers everything. Alerts fail too.

8. Reports provide inaccurate information.

And I can go on and on and on and write till I reach character limits. The issues all seem compounding each other in that you fix one and create another 20 in the process. I'm almost convinced best thing is to drop everything and start fresh. But then question becomes how can we preserve current historical data?

So what do you think? could it get worst? and if not is there hope for a brighter day? hahaha.

Find more posts tagged with

help

fialing

not working

unstable

npm - network performance monitor

problems

npm

broken

failing

issues

Accepted answers

All comments

rschroeder

Yes, it COULD get worse.

Before that happens, run the Diagnostics Wizard on all of your instances and polling engines. While those services are running, open an online support case with SolarWinds for your problems.

It sounds like overkill, but I suggest opening eight separate cases, one per specific instance. You'll probably get better results that way, and you can reference all eight tickets as you work with the Tech Support staff; maybe they'll see a pattern and combine some of those issues into fewer cases.

Once the online cases are created, attached the diagnostics to each.

After the diagnostics finish uploading, call SolarWinds Support and reference the number of the first open case on which you wish to work. Then start knocking them out one at a time. Yes, it may take you a month. That's unfortunate. But you'll end up with a product suite that works. And that's what we all want, isn't it?

Good luck! Let us know what the solution is for each of those issues. I'm certain others will be watching this to see your success to learn from your experience.

Swift Packets!

Rick Schroeder

lcsw2013

I greatly appreciate your response. This may or maybe not make you laugh a bit but this has been done already. Let me explain. When I took over this environment 3 years, the environment was in even worst shape. I took one look at it and said this infrastructure will crash in 2 weeks. The Monday after, guess what, it crashed. haha

I asked to rebuild fresh. Had strong push back and similar push back for updates. And after another 2 crashes was when I got approval to update. We updated but the environment wasn't up to snuff at that point. Servers all out of spec, performance issues, etc.etc. So through another act of congress I went to finally get better servers. We rebuilt fresh. Applied all the KB's support provided. Followed all the rules. Problem is we carried the problems from previous environment to new.

We average around 12 tickets per month to support, We've spoken to developers, we've spoken to application engineers, with sales engineers, I think we've spoke with product managers. Had plenty of sales escalations etc. Got to a point where we even had a large meeting with management here and management in solarwinds.

Even after 3 years of effortless hammering away we've only improved about 3 to 4 percent from were we where. Support only pushes upgrades. Because issues are drawn out and take so long to fix by the time we get close to a fix we are told to upgrade with promises that issues will be fix. Turns out they never are.

We've even engaged loop 1. They provided tremendous help. But still that barley move the ticket on progress. This is why I was toying with the idea of starting 110% clean and fresh. Maybe storing historical on a dev side and starting a new. Only way I can see the light at the end of the tunnel.

But you know, curiosity struck me and I was thinking I couldn't be the only administrator out there with so many pain points. Let me see what my fellow admin's on thwack had to say. Thus we are here. .. I have a ticket currently with them but not holding my breath because it seems to be going the usual route of them suggesting an upgrade which we will not do until we have solid evidence and proof of problems being fixed.

Thanks again man! I appreciate it. And I will need all the luck I can get hahahahaha

rschroeder

I'm sorry you and your organization are having so many problems with SolarWinds products. I appreciate your idea of starting from scratch. But you have a lot of things that are good, right? At least you're getting alerts from NPM when systems or sites go down, aren't you? Or are you relying on users' complaints and help desk calls to tell you what's not working?

Let's imagine you don't use my first recommendation of having SolarWinds Support help you out. ding and serena and DanielleH know people and departments who can help you get that part straightened out. Maybe they'll read your REALLY sad story and try to escalate some additional attention your way on your open case(s). I think you should follow that path, using the steps I suggested.

But, if you were to go for a complete wipe of your system, I'd still go with what SolarWinds Support recommends--upgrading to the latest versions and applying the appropriate hotfixes. It's really the right way to eliminate old problems, to build cleanly.

So you'd do that. Get Support's help to wipe out old and replace with new. Maybe you'd create VM SnapShots of all the old servers for insurance.

Support should be able to help you export what needs to be exported for a fast and efficient new deployment. They may recommend you change hardware, upgrading to something with more memory or faster CPU or more processors. But they'll get a good outline of what must be done to minimize your NMS solution.

Your original 8 steps, and what I think of them--IF you don't work with SolarWinds Support (you really should do that):

Orphaned phantom service. I wouldn't worry about it if I were installing new. If I was upgrading, I'd fix it, hot fix it, patch it, or start from scratch. I assume this is on your main instance of NPM, right? Have you tried running the Configuration Wizard to fix it? Replacing recommended files? Rebuilding recommended files? Support REALLY knows their stuff here--I wouldn't ignore their recommendations. But if you must, you COULD simply blow that server away and build a new NPM main instance. IF you blew the old one away (after making a snapshot of it) you could even use its original IP address, which would be simpler in many ways--especially if you have multiple APE's and if you have systems sending traps and syslogs and NetFlow data to that address.
Primary poller opens 12,000 WMI connections to unauthorized destinations? That sure sounds like malware or a Trojan or a Wyrm. I'd verify that system isn't compromised; run the anti-virus, patch it per Windows and SolarWinds recommendations, etc. Or again, blow it away after making a snapshot and build new, using the original IP address.
Communications dropping from agents. That sounds like a known problem that I recently read about in Thwack, and the fix was upgrading to NPM 12.3.
Poor http rendering on the main site. Since you've rebuilt it 30 times and run and rerun the Config Wizard, what did Support recommend? If you haven't talked with them about this as a single specific issue, you're missing out on a great resource. Of course they'll ask you to upgrade to the current version--why would they not? The new version has loads of problems that are fixed, that are designed out of existence, plus new patches. Get it. Or, blow it all away and start from scratch. But please talk with Solarwinds Support. They might take a few weeks, but they rarely will steer you wrong. And you can always escalate to higher level for better help if you're unsatisfied with the progress you're receiving.
Database timeouts. I talk with my in-house DBA's about this--they're great resources. You might even reach out to aLTeReGo or sqlrockstar for ideas and things to try. But they'll (reasonably) ask what SolarWinds Support has done for you on this. If you haven't touched based with Support on this as a specific issue, you might be fighting something so low-level that you must fix the underlying problem before you can address the actual one you see here. At worse, what's the penalty for losing all of your historical data? You've lost some trends. You may lose your NCM backups. So copy them elsewhere if you're going to build a new database on new hardware. But see to it that the new environment is truly meeting SolarWinds guidelines and specifications. Otherwise you may be nickel and diming yourself right back into a pit.
Blow IPAM away. Delete it, reinstall it, if you've tried everything else (including a separate ticket with SolarWinds Support to troubleshoot it). Hopefully it's not your sole source of DHCP addressing and services, Fixed/Reserved/Available addresses. If it IS your only point of DHCP management, don't blow it away without creating an export of everything that can be re-imported into a new solution. In that case I'd definitely use a snapshot and back it up before I touched it. In fact, I'd use an entirely different machine for the new install, and shut the old machine down, in case you had to quickly go back to the old (bad) IPAM environment for production reasons. But since it isn't working, it sounds like you don't "need" IPAM to function--it's not your only / main source of DHCP management, is it?
Alerts being weird. This sounds like corruption or a bug. I'd expect a hotfix or a new version to completely fix it. What did Support say about this specific problem? You have so many issues that it's going to be a challenge making progress on them all simultaneously, so once you have a dedicated support ticket built for this sole issue, SolarWinds can focus on it specifically and help get it resolved. Or, blowing it all away and rebuilding a new solution (I don't recommend this path) can remain an option.
Reports are inaccurate. This sure sounds like it's tied into the database problems mentioned above. I suspect once you have that remedied it will fix MANY of the reported issues.
Preserving historical data. What happens if you lose it? Personally, although I have nearly 70,000 active devices on my network, spread across a hundred hospitals, clinics, and business offices, no one would be hurt if we lost our historical data. As long as I have a list of all nodes, their IP addresses, credentials, user names, passwords, TACACS/RADIUS hosts & pre-shared keys, snmp community strings, sites, locations, etc. I can rebuild everything manually if I had to. And still, no one would be hurt. Especially if I have electronic copies of all of the startup & running configs on all the devices. All of this can be used in a new Discovery on a rebuilt / replaced system. So, how painful is it to you or your company if you don't preserve the historical data? You'll start building new historical data the second you begin polling. And shortly thereafter you'll have a day, then a week, a month, and a year's worth of data. How crucial is it to lose the old data?

There IS hope. But take this in baby steps. Once you break it down into workable segments, you've begun the triage. Then you can focus on the worst problem, or the thing that needs to be fixed first to get the most return on your time, to get the most results, to get the most systems responding in helpful fashion.

Don't give up or give in. DO give Support a call. And reach out to some of the others I've mentioned. There are SUPER people out there, some of whom will have ideas about solutions based on what you've provided.

Good luck, and Swift Packets!

Rick Schroeder

mesverrum

Leandro, honestly it sounds like a brutal mess but I would also second the idea of starting fresh, hosting a 'historical' copy of data for anyone who needs it, and migrating everything over. I'm going to jump into a thread with you and Doug Keyser when I get back on the ground this weekend. It pains me to see how you've struggled and I think I can give Doug whatever guidance he might need to be able to get you stood up in a new instance with as seamless a transition as possible. I suspect that I have more "moving to a new instance" scripts than anyone around so I'll try to coordinate with you guys there. I'm kind of looking forward to the challenge of recovering what you guys have.

mesverrum

For a point of reference, I'm working this week with a client running over 40 ape licenses, 160k interfaces, monitoring over 10k Sam apps and a mountain of components. Their system does take some babysitting now and then but the team handles it along with several other tools and custom integrations and it keeps up and serves the purpose keeping an eye on all that. If they can take orion that far beyond the "rated" limits then I don't see why we wouldn't be able to wrestle your environment into shape.

hebron1212

Were you able to preserve your historical data? If so, how?