When Pingdom selects a server to confirm downtime, it should avoid servers from the same country. Different datacenters and ISPs are not enough.
Recently, as the main communication route between Japan (where our servers are) and Australia was experiencing issues, when a probe server in Sydney could not connect, Pingdom repeatedly selected the other probe in Sydney for the second opinion. The 2nd probe may have been in a different datacenter and using a different ISP, it was geographically too close to the original probe, hence was also affected by the regional traffic issues. This lead to repeated false positives, triggering our incident response.
Note that as per the documentation, Pingdom should already be trying to make the second opinion as geographically different as possible:
> When one of our probe servers cannot connect to a site or server, Pingdom's system will first mark the check as Unconfirmed Down and then ask another probe server to try to make the same connection, we call this a Second Opinion, we try to make the second opinion as geographically different as possible to make it easier to determine where the issue is.
But given this recent incident, there is room for improvements.
