Reduce or nullify solarwinds upgrade downtime?

Good day, I am getting almost daily flak for our (albiet infrequent) solarwinds upgrade windows. Since we have grown to 8 APEs, the duration of the upgrade window continues to grow. Our last one one lasted almost 5 hours due to some Configuration Wizard errors. While I understand other customers ( a major sporting goods retailer) have upgrade hits of up to 10 hours, what alternatives do I have to continue to monitor my environment while the window occurs?

We stage our upgrades during the day so the business can view their applications, and Wintel, Network, and UNIX teams can use back up systems to view for downtime and issues...this cuts into their already busy schedules.

1. Take a hit, get justified flak from management?

2. Buy another disparate solarwinds system at a cost of upwards of hundreds of thousands of dollars?

3. Add an additional server, and database; and monitor with ICMP for up/downtime for at minumum Critical servers?

It would be great if solarwinds provided free 24 hour licenses for above. A temp enviroment, as it were...

4. Other ideas? Am I missing something?

5. Yes I know NPM 12 is easier to upgrade...

Thanks!

Find more posts tagged with

ncm upgrade

centralized upgrade

orion installer

zdu

rage against

npm upgrade

upgrade_advisor

solarwinds upgrade

Status: None

Comments

familyofcrowes

I do feel your pain on this. WE have 6APE's, additional web server, WPM server, NTA DB server and they all must be touched. Then when you take into consideration that we have many modules, SAM, NPM, NCM, WPM, NTA, UDT, VNQM, IPAM and Toolset, the install/upgrade takes forever.

We now do more upgrades with max of two modules at a time to reduce the downtime. We cannot afford the downtime.

Traps are some of the biggest issues because if the system is down they are gone forever. We have many critical systems that alert via traps.

I have no easy suggestions. But there are engineers much smarter then I at Solarwinds that I'm sure are looking at this....

BTW: you can get 30 day evals that work 100%. But that takes alot of time to get that all setup, and I'm sure you have a large DB that can complicate things

planglois

If at the very least there were a "Maintenance mode" like .

Looking forward to upgrade to SAM 6.3 to finally test the new Centralized source update feature.

rschroeder

I had an encouraging conversation with SW staff about this topic earlier this week. While they would not specify what may or may not be changing in this realm, or when it might or might not occur, I felt optimistic after sharing my frustrations about upgrades with them. They had good things to say.

I feel something new is not that far away, and it might address this very topic in a positive way. Hope springs eternal!

smoked_angus

This in a way is encouraging, thanks for sharing sir.

planglois

Syslog & trap alert rules should continue to work while the upgrade is done (or at the very least, messages should be buffered)

familyofcrowes

I'll be upgrading 5 modules next week....

downtime kills us..... We are a 24 hour shop and downtime costs us $$

so.... bump this babyUP!

johan

We have 3 primary systems and +- 16 Additional pollers in a MSP setup. I am definitely in agreement to this as I also get beaten up daily on our downtime slots when doing upgrades. Trying to co-ordinate downtime slots between 80+ customers spread across continents and time zones is also not something that is easily achievable.

PLEASE Solarwinds - make this your #1 priority!!!

I vote YES YES YES YES

monitoringlife

30 SolarWinds servers total. MPE and 13 APEs. Everything is in HA so doubling base number. 2 web servers.

Servers with high latency (on other side of the planet) timeout. Centralized Upgrades have to be stopped and have to be done manually.

1. Take a hit, get justified flak from management?

This is what we currently do.

2. Buy another disparate solarwinds system at a cost of upwards of hundreds of thousands of dollars?

I don't think I could justify that. The additional administration time keeping two instances would be excessive for me.

3. Add an additional server, and database; and monitor with ICMP for up/downtime for at minumum Critical servers?

I think my security teams already find me taxing with the number of ports and networks I need to transverse. Having to maintain an extra system and extra APE's to get to secure networks would add administration overhead.

It would be great if solarwinds provided free 24 hour licenses for above. A temp enviroment, as it were...
Would be cool. I think the older versions of SolarWinds had a companion binary that ran ICMP checks while the platofrm was offline doing upgrades?

4. Other ideas? Am I missing something?

5. Yes I know NPM 12 is easier to upgrade...

Centralized upgrades is amazing, but it does not like high latency sites. I would love the ability to push out the binary files in advance of the upgrade process being executed.

It would be cool if the upgrade could work on the HA nodes and allow us to maintain uptime.

MathieuJM

We have MPE in HA, 3 APE in HA and 10 APE without HA and of course we have the same questions about the upgrade process.

My management wish to have a zero/near zero downtime or at least to have a vision on the infrastructure during the upgrade.

We have the same question :

- use another monitoring solution (SolarWinds or not) however this will have cost (OS, licence, CPU/RAM/DISK ressource, time to keep it up to date) and of course the data are not saved in SolarWinds DB,

- use a kind of distributed ping tool (executed from the MPE/APE so that there is no need to create new firewall rules) just to test the network availability (again no data in SolarWinds DB)

- any advice are more than welcome

designerfx

familyofcrowes don't forget to run Orion Upgrade Buddy on each poller while upgrading - . It can at least monitor everything on each poller while you're upgrading.