Community
- Command Central
- MVP Program
- Monthly Mission
- Blogs
- Groups
- Events
- Media Vault
Products
- Observability
- Network Management
- Application Management
- IT Security
- IT Service Management
- System Management
- Database Management
Content Exchange
- SolarWinds Platform
- Server & Application Monitor
- Database Performance Analyzer
- Server Configuration Monitor
- Network Performance Monitor
- Network Configuration Manager
- SQL Sentry
- Web Help Desk
Free Tools & Trials
Store

Building a Dry run environment for Solarwinds to test upgrades

I never get a smooth experience while performing upgrade. So I want to have a Test environment to test upgrades before going live. Of course this means, the test environment should be able to poll our production equipment and receive SNMP traps, syslog from it.

I cannot perform upgrade on Failover Engine to test, because the ugprade would have still modified the Database and hence irreversible. So it seems, there should be a separate set of App & DB server to do this, with production data dumped in to the DB prior to testing. Also it would be better if the poller retains the same IP address so that we don't need to reconfigure devices (which of course means hiding one of the two environments at any point of time)

How do you guys go about doing this?

Find more posts tagged with

Accepted answers

RichardLetts

Oh my!

There really should be an architectural diagram on how FOE works. What it does: it replicates the files, directories, and registry settings that contain configuration information. It does not replicate other registry settings, service installs, binaries, etc. So, you upgrade the active servers, then switch active servers, and upgrade the (newly) active servers. this gets your configuration files, etc. upgraded and then replicated.

Solarwinds have a document on how to perform upgrades in a FOE environment:

SolarWinds Knowledge Base :: How to Upgrade SolarWinds Orion when Orion Failover Engine is Installed

It works; you can run for a while at step '4d' to verify the new applications are working and then proceed from there

If you have to perform a roll-back then after 5.c.i you should restore the database and re-install the old versions on your secondary servers.

when we initially setup FOE I had a lot of trouble getting it stable and testing out failover scenarios. It has been rock-solid since, and every month we test the failover when we apply the operations system patches using this document:

How to Apply Operating System (OS) Patches or Hotfixes with Minimal Interruptions

It was so difficult to get FOE setup in the first place, that I'd avoid trying to uninstall it and then do the upgrade

Aside: FOE and NCM 7.1 additional pollers are incompatible.

All comments

adatole

The Bad News: The only answer is to buy a second copy of Solarwinds and install it on a development/test/sandbox machine.

The Good(ish) News: SolarWinds will sell you a "lab" license at 50% of the list price for a full license.

What I typically do is:

1) Buy the licenses for my production environment

2) Buy the SMALLEST license level for the same components

So if I have a client with NPM and SAM (all unlimited) along with additional polling engines and an additional web server, I would buy a "lab" setup of NPM SL150 and SAM SL150 (the smallest they sell, IIRC), which I get for 50% of list price and use that for development, testing, patch testing, etc.

If I just want to play around with something (a new version, a module I don't already own, etc) then I build a VM and install the 30 day version to play around. If it passes that test, I'll then install it in dev.

Hope that makes sense.

thamizh85

It does, but how do you make it working in a production environment? In our setup, if a new Monitoring station needs to manage our devices, lot of changes need to be done such as updating SNMP ACL's, Firewall rules, Syslog hosts, snmp-server hosts etc.,

Testing it over a few local devices is also not satisfactory, since some of the upgrade decisions pending on management review want to look at wholistic view provided by the NMS (Say hardware health monitoring in 10.4, our management would like to see how the dashboard looks on our environment)

I am thinking about a monitoring dry run environment, where I get to use the same hostname and IP address as production systems, hide the Production setup from network, run the dry run environment for few hours or days and be able to fall back to the old Solarwinds Production setup.

RichardLetts

If you have a complete FOE setup (including a replicated database), then the way to do this is to

a) break the database replication (effectively keeping a backup)

b) turn off one complete set of servers and perform the upgrade on the running set.

If everything looks good turn on the (old) servers re-establish the database replication and perform an upgrade on the app/polling servers

if it does not look good then back-rev your servers, point the servers at the old database; resync the 'good' database to the 'bad' database (and consider your options). You'll lose stats collected for the time-period you were running on the failed-upgrade version.

I think this is the intent of the upgrade instructions as provided by solarwinds, except they don't go into detail about database mirroring since they are not responsible for that, instead relying on backup/restore of the database to enable roll-back.

m_roberts

If I was advising one of our clients I would suggest that going towards Adatoles suggestion.

Putting a small sized license in place if you need regular access to a test lab or each time you want to dev test install a trial version. With both options create a copy of your production database to use as the foundation. Most QA testing can be achieved on a subset of devices, so could you create a rule in a segment of your network for an additional monitoring server for access?

Going down the route of mirroring the identity of the test server is only going to complicate matter in terms of procedures and increase the chances of loss of data.

Please post back how you get on, it would be good to hear your experiences if you do go down this route.

thamizh85

We do have a complete FOE setup, but so far it has only complicated our upgrade process. We recently performed a major upgrade and hired some professional service to execute it. As per their advise, we shutdown the old FoE standby nodes and proceeded to upgrade the active nodes. And they advised to build new VM's and rebuild the FOE with the new VM. But their process failed at rebuilding the FOE and now we are stuck with an environment without FOE and pending on vendor to work out the setting up FOE again. I would work on it myself if only I had time for it and also not keen on it, having paid for professional services.

OK, the reason I am quoting this story is, I am not sure what is the best way to upgrade with FOE. There seems to be 3 ways to do it:

1. Upgrade on active node and bring up the standby node, resync the file system and registries (would that suffice to upgrade standby node?)

2. Upgrade on active node, do the testing, then hide the active node, upgrade standby node and resync both (Is this what you mentioned in your reply?)

3. Upgrade on active node, setup a fresh vm and rebuild FOE (as suggested by our vendor)

If it is workable, option 1 or 2 seems to to be the best way to test newer versions before rolling out. Of course one needs to get quite comfortable with switching between FOE, rolling back and stuff. So far, FOE seems too troublesome to play with.