SolarWinds Failover Option With FoE at a Different Site with Different IP

Hi Guys,

I am just hoping to get some feedback on Failover option whereby FoE is located in different physical location with different IP address. I went through documentation (link below) and would really appreciate your help on the below:

1. Is anyone able to elaborate in simple terms on steps involved to Failover and where does this public IP fits in (as per screenshot below it says public IP is 192.168.1.5 but on the image itself it is not shown)?

page 14, quote:

When configured for Disaster Recovery (DR) to a remote site with a different subnet, SolarWinds Failover Engine must be configured to use a different Principal (Public) IP address for the primary and secondary servers. When a switchover is performed, the DNS server will be updated to redirect users to the new active server at the DR site. <..> SolarWinds Failover Engine uses DNS Update task to update Microsoft Windows 2003, 2008, 2008 R2, and 2012 DNS servers with the new Principal (Public) IP address.

2. Some nodes will be sending Traps, SysLogs and NetFlow to Poller's IP at the main site. What would be our possible options for redirecting this traffic to another site without having to update config on each of them in the event of failover?

3. And last one... What are ways to ensure high availability for NetFlow database. With SQL database it is all pretty clear (mirror, logs shipping, cluster, etc), but how do I ensure failover of NetFlow Database AND what steps will be necessary to actually fail it over in the event of disaster?

Any help is much appreciated

Thanks a lot

Alex

Resources:

http://www.solarwinds.com/documentation/failoverengine/docs/OrionFailoverEngineAdministratorGuide.pdf

Comparison - Orion High Availability vs Active-Active.pdf

Find more posts tagged with

failover engine

foe

Failover

Accepted answers

blsanner

I can help shed some light on this as I have done several FOE deployments.

1) In simple terms, with FOE, you usually have two network interfaces on each server. One serves as the public interface and this is the interface that handles the normal server communication such as polling, website, database, etc. The second interface is the heartbeat channel which is used by FOE for synchronization and communication between the two servers in the pair. Now, the magic of FOE comes into play with a packet filter which gets installed on the public interface on both servers. Since the servers are clones of each other all the way down to the SID, both can't be talking on the public network at the same time. In normal operation, the packet filter solves this by disabling communication on the public interface of the secondary server. When a failover occurs, the packet filter is enabled on the primary server (if it is still up) and disabled on the secondary which allows the secondary to take over the polling, website, etc. duties. In a WAN DR situation as you are describing, it goes a step further and changes the DNS record for the server to point to the IP address of the secondary server as well.

2) As you mentioned above, this is not easily done. The only thing I've seen concerning this is to use a load balancer and point these items at a virtual IP and let the load balancer handle where the data actually gets sent. This way you only need to change it in one place. But, in reality, if this is truly a DR solution, while these would be a nice to have, I think you would probably have bigger concerns.

3) As for the Netflow storage database, since it is just stored in files on disk drive, you could use any solution that would account for that whether it be a cluster or some other file/directory replication solution.

Also, as for the question about the static routes, that is needed in the situation where the two heartbeat interface IPs are in separate subnets. You should only have one default gateway on a Windows server and, with the packet filter enabled, the heartbeat traffic won't be able to use that default gateway. A static route will tell the server to use the heartbeat interface to reach the other server in the pair using the heartbeat channel. I've also seen people set up stretched subnets across their sites. In that case, a static route would not be needed because the two heartbeat IPs would be in the same subnet.

I hope that helps.

- blsanner

Loop1 Systems: SolarWinds Training and Professional Services

LinkedIN: Loop1 Systems
Facebook: Loop1 Systems
Twitter: @Loop1Systems

All comments

HerrDoktor

I assume this is a typing error. The Principal Pubilc IP should be 192.168.1.127. As stated in the text:

the Principal (Public) NIC on the passive server uses the same IP address as the active server but is prevented from communicating with the live network through an IP packet filtering system

As the only 2 identical addresses are 192.168.1.127 I'd say this is it. Other than that I can not think of any way how you would configure this differently.

To be honest, I had my nightmares with FoE and I tossed it...this is not a good way of implementing HA. I think there is a Thwack-idea to be able to use Microsoft Failover Cluster for Solarwinds. You might want to vote for that.

Cheers,

Holger

----

Edit: There is something in the "What we are working on" Section already. There is a sneak peek in SAM 6.3 when I remember correctly

Check Here FoE - Help us Help You

AlexSoul

Thanks Holger, I had a feeling that this might be typo ... appreciate your feedback but I think I will give it a go, considering it is Disaster Recover that I am after and that it is supported solution

Well, I found a great article by bshopp here and after reading it it seems like #2 might not be easily achievable - can anyone else comment from their experience?

Also, BillFitz_Loop1 is mentioning in the same article the necessity of adding static routes and I haven't found anything about it in documentation - is it really requirement? If so, where can I find doc that talks about it? ... I hope there should be step-by-step guide somewhere to configure Disaster Recover FoE option over WAN

AlexSoul

aLTeReGo - hope for your help on this one as well (we are in design stage, didn't use FoE before). Thanks a million

blsanner

I can help shed some light on this as I have done several FOE deployments.

I hope that helps.

- blsanner

Loop1 Systems: SolarWinds Training and Professional Services

LinkedIN: Loop1 Systems
Facebook: Loop1 Systems
Twitter: @Loop1Systems

shuth

For 2 - If you don't mind duplicate traffic on the network (depending on how much data you're sending in traps/syslog/NetFlow), you could configure your devices to send this data to both endpoints. Only one endpoint at a time will actually be listening as the interface will be blocked and the services shut down on the second endpoint.

rschroeder

This is an interesting complex project, of which I was unaware.

When I read that it relies on automatic updates to DNS during a failover, and that polling relies on DNS instead of plain IP addresses, I began understanding the design.

I'm impressed, but haven't tried it--my environment hasn't place that degree of importance on Orion monitoring to justify the work needed to be done. But I thank you for posting this interesting situation--I've benefited from reading the comments.

AlexSoul

wow, blsanner thank you so much for your input and time invested to clarify. Really appreciate your help

... looks like we are good to go then I will review documentation with a fresh eyes now. Will post progress here...

Thank you so much for your help and time,

Alex

AlexSoul

blsanner, may I please disturb you with another question:

How do you manage FoE offline server? i.e updates, software patches, AntiVirus, domain membership, etc - or this all becomes irrelevant and FoE app does replicate all these changes automatically? ... including AntiVirus for example?

Thanks,

Alex

sonic9t9

IMO you would want to think of a fail over solution for the whole data center and just not this product. May want to think about MPLS type connection or l2 extension depends on money and all sorts of other things.

rschroeder

We do have other failover solutions in place, but only the Network Analysts rely on Solarwinds, therefore it's not mission critical, and a failover solution is unnecessary.

The thought is that if Orion's down, we've lost our ability to discover the extent of other down systems, but we can restore Orion and regain that visibility. If a data center or all of its resilient BGP links in/out were down, and if that happened to be the reason NPM and NCM were down, we've got bigger issues to deal with than NPM failover solutions.

sonic9t9

True but being that solarwinds is likely to be in the datacenter. It should be part of a plan to restore systems. Also, other things likely tell you systems are down aka vmware.

blsanner

No, these don't replicate automatically. Solarwinds provides documented procedures both for updating Solarwinds Orion itself as well as installing operating system patches:

Upgrade SolarWinds Orion when Orion Failover Engine is Installed - SolarWinds Worldwide, LLC. Help and Support

Apply operating system (OS) patches or hotfixes on FOE with minimal interruptions - SolarWinds Worldwide, LLC. Help and …

AlexSoul

Hi Guys, just quick Heads Up - I got news that SolarWinds has stopped selling FoE. I am not quite sure it has been officially published already, but we will have a call with support to brainstorm other alternative options. Looks like FoE is not working out and after digging through documentation I would incline to agree with rschroeder for the fact that it may well end up being "liability" rather than "asset"

Thanks everyone for contribution to this thread - always great to learn from masters here

Thank You,

Alex

blsanner

If you look at their new offerings, this should not come as a surprise. They now have HA built into the Orion suite itself starting with SAM 6.3:

SolarWinds Orion 2016.2 feature: High Availability - SolarWinds Worldwide, LLC. Help and Support

shuth

The difference between the new HA and FOE is that HA only supports servers in the same subnet. If you need HA across datacenters, you're going to need to span layer 2 between them or use FOE.

Or run up a second standalone system but that means more licenses and extra admin work.

sonic9t9

layer 2 or mpls type extension. or with nexus I believe you achieve it through otv option. l2 can be done a few different ways too. like we use our dwdm to extend it

AlexSoul

Exactly right shuth - spot on! ... and this is what we are facing with. Network team is very reluctant stretching subnet across sites. So, we are left with exactly 1 choice then - Active-Active

Additional resource:

I have attached pdf we have received Today from SolarWinds support (above, in the main question, as I could not attach to the reply here) - "Comparison - Orion High Availability vs Active-Active.pdf"

AlexSoul

Hi guys, I have just publish a new thread on the alternative for this FoE solution. It is in RFC/Design stage and I would highly appreciate your feedback, ideas, concerns and thoughts - maybe you will spot something I have missed. If we will nail this one down - it can well server as an interim model for DR/HA solution, at least until SolarWinds come up with HA for inter-site without the need to have HA cluster sitting on the same subnet

High Availability and Disaster Recovery Solution with Full Servers and Site Protection - RFC/Design Stage

Your time investment is much appreciated,

Alex