FoE updates the DNS entry almost immediately. What you are likely encountering is DNS caching on your local machine. You can test this yourself by performing a failover and then running the 'ipconfig /flushdns' on your local workstation. Provided you are using the same DNS server the one which was updated by FoE, the change you see would virtually immediate. If you are using a different DNS server than the one that FoE is updating, then your DNS server is also likely caching.
This is perfectly normal behavior for any DNS server, and essentially any client operating system to cache DNS results. You can however control how long these results are allowed to remain cached by adjusting the TTL for the DNS entry. This would be a value controlled on the authoritative name server for the domain.
Maybe our problem is not that FoE doesn't update one of the dns servers quick enough but the fact that dns doesn't replicate between servers quick enough.
When we fail our main app server over, the website is down from anywhere between 5 to 30 minutes. That's if one connects via the hostname, if I connect to the IP address of the now actice secondary server the webpage works. Would there be something that can be changed in dns so that the new/or altered dns entry replicates out to all the other dns servers immediately?
Yes -- ask your DNS engineers about this.
1. Make sure the TTL on the resource records is set to 30 seconds
- this should mean that the results of DNS lookup are cached for this long
2. Set the REFRESH on the SOA record to ~30 seconds
- secondary DNS servers will check with the primary DNS server to see if the zone serial number has changed this frequently.
- DNS engineers may not like this
In my world our Solarwinds servers are [currently] in their own zone that I control [I'm also one of the DNS engineers (I have too many hats)] with a short keepalive
Shortly we're hoping to completely stop using DNS updates and switch to DNS load-balancing where the DNS servers maintain a keepalive to the Solarwinds Servers through the FOE-managed interface and will return the IP address of the server currently working (saves us doing a DNSupdates from FOE, and also gives us geographic redundancy).