Tested Execute Alerts - Not Showing Correct Date/Time

So weird question, but has anyone encountered a situation where the alert time doesn't match up when doing an executed test on an email action?

Here is the scenario I am running in to:

Doing some alert testing with a client. I select the device I want to test and execute the email action. The email sends successfully, but the date/time on the test is not current. No matter what device I test, it does the same thing - the month is usually off by a month or two and the time is entirely random.

The weird part:

Live alerts for these devices show the correct date/time. A test I recently did: ran an executed test on an interface and received the test email with the date and time of October 21st, 2021. Purposely brought the interface down to trigger a real-world alert on Tuesday and received the email stating November 30th, 2021.

Is the random date part of the design on the executed tests? I usually just do simulate when designing my alerts, so I never would see the date/time, so it's been quite a head scratcher for me!

Find more posts tagged with

Alert

email alert

question

Alerting

Accepted answers

All comments

gombosi

Forgot to mention, I checked the alert logs to see if its simply picking up the date for when the interface previously alerted, but I have nothing for October 21st. Like I said, a head scratcher.

shuth

What is the variable you are using in the alert? This should show the trigger time of the alert:

Alert Trigger Time: ${N=Alerting;M=AlertTriggerTime;F=DateTime}

When you simulate/execute the email action from the alert configuration, it doesn't input any datetime as the alert doesn't actually exist. e.g. mine shows: "Alert Trigger Time: Never"

gombosi

This is what I have in the subject

ALERT: Interface ${N=SwisEntity;M=FullName} on ${N=SwisEntity;M=Node.Caption} is down at ${N=Alerting;M=AlertTriggerTime;F=DateTime}

EDIT: See my reply below for the copy of the test email I recieved.

gombosi

This is the executed test message I received:

ALERT: Interface [location removed]-mx960-b-re0 - et-9/1/0 · 200G LAG Link#2 to [location removed]-mx960-b ae9 et-5/3/0 - [location removed]-8800-Trib C:6-1 on [location removed]-mx960-b-re0 is down at Sunday, October 31, 2021 8:04 PM

Here is a screenshot of the email:

shuth

The test/simulate won't have a real trigger time as it's a fake alert. I've never seen it say anything other than Never in an execution/simulation though so not sure why yours is showing October.

I'd suggest copying the alert and modifying the trigger conditions to a known condition - i.e. that specific interface and condition = up, or the custom property "Comments" of an interface is equal to 'test trigger alert'. Then cause that condition to trigger a real alert.

You can also input the other datetime fields in the alert message to confirm your server knows the current date/time. e.g. ${N=Generic;M=Time} or ${N=Generic;M=DateTime;F=DateTime}

Other than that, the usual suspects - confirm your SolarWinds server and SQL database are in the same timezone, correct date/time, etc.

gombosi

All of the servers have been confirmed to be on the correct/same date and time already, so unfortunately its nothing simple like that. I'd have to imagine that the server knows the correct date/time since any time the client receives a real-world alert on their devices, the emails they receive have the current date/time, but I'll try those date and time fields you mentioned to see if anything changes.

Thanks!

gombosi

I ran a test and got the following:

So the first two Generic codes show the correct time, but the one using N=Alerting;M=AlertTriggerTime comes up with a completely random incorrect time.

JaroslawLadyga

I can add that the time the alert triggered may differs from time you receive alert message for some reasons like alert escalation (repeated) action, alert delay, mail system issues etc. And more - time when interface went down is not the same time when alert triggered. It depends on alert conditions and settings.

gombosi

Unfortunately, I've been wrestling with this for roughly two months on and off - so I've had to rule out a number of issues already. The client's mail server appears to be a non-issue, the alert is set to trigger if a condition exists longer than 2 minutes (critical devices) - so delays shouldn't be a heavy issue and the alert itself is written very basic so I don't think the conditions are the issue (basically what you would see in a basic interface down alert).

I'd imagine that if it were an influencing issue like a condition or external mail server misconfiguration, we would see issues on all DateTime conditions (like N=Generic;M=DateTime), but it seems like it only messes up on (N=Alerting;M=AlertTriggerTime). I'm no SQL/SWQL expert by any means, but can that suggest an issue with the table? Just trying to think what else to check at this point.

sum_giais

If the only time you see this inconsistency is when you test (via editing the email action and select simulate / execute -- there's the simple fact that the alert trigger time is just that.. an alert trigger time.

The issue really comes down to the fact that it's an alert variable (alert trigger time) and when you test simulate or execute the action (note action.. not the alert) there's no alert record to pull that variable data from.

So end result is there's no alert record, hence unexpected results. Its kind of a moot point if its not being seen in the actual alerts that trigger.

As others have recommended - try actually testing a real world situation that truly triggers the alert like bringing an interface down. Otherwise your just testing the assigned alert action.

I also notice that from your screenshot, you're going to the actions tab and selecting test on the action. From that context / view your test is not explicitly tied to any alert definition. I've seen strange behaviour when testing it this way so I make a habit if I need to test an action I go to edit the alert definition, navigate to the trigger actions or reset actions as needed and test simulate / execute from there.

Try doing that - go edit your alert definition you want to test against and navigate to the trigger actions tab, then test execute / simulate there and see if there's any difference.

Venturing a guess it may be pulling the last trigger of any alert against random alert definitions for these entities (interfaces?)

I hope this helps -

KMSigma.SWI

One other thing that might help us out: export the alert from your system and attach the XML file. Then some of us can import it and look at all the logic.

saw10

Did you ever get this sorted?
I have found that the alert time used is how your alert is "written" -
Direct pull from database (SWQL/SQL Query)
VS
Alert Creation via the "Alert Manager"

SWQL/SQL - this will use the "raw" database information and timestamp - The database uses UCT as it timestamp.
Which is then modified with the timezone setting of the primary poller, which is used to to display on your Orion website.
Alert Creation also uses the modified timestamp. The timezone of the primary poller.

I may be totally wrong but I have found this to be true for my alerting.
Something to think about.

gombosi

Thanks for the follow up! We ended up changing the alerts to N=Generic;M=DateTime and so far that appears to be working for this client.

jacksonliam341

I select the device I want to test and execute the email action.

The email sends successfully, but the date/time on the test is not current.

No matter what device I test, it does the same thing - the month is usually off by a month or two and the time is entirely random.

Live alerts for these devices show the correct date/time.

A test I recently did: ran an executed test on an interface and received the test email with the date and time of October 21st, 2021.

Purposely brought the interface down to trigger a real-world alert on Tuesday and received the email stating November 30th, 2021.