Pingdom alert is being sent immediately even when "When down, alert after" is set to 5 minutes

I have the "When down, alert after" setting set to 5 minutes for my pingdom check.  However, it sends alerts immediately.  Do I have this configured incorrectly or is something else wrong?