cancel
Showing results for 
Search instead for 
Did you mean: 
Create Post

Acknowledge Solarwinds alert by email

Updated 2019-02-26 to support recent changes to the Solarwinds API.

Updated 2019-07-02 to support passing credentials as encrypted credentials files or as clear text username and password on the command line. To keep Task Scheduler configuration simple, I've included instructions for setting the credentials within the script.

This Powershell script allows end users to acknowledge or comment upon a Solarwinds alert via email. The script assumes that your organization uses Exchange as its email platform.

Prerequisites:

  • An email account ("the email account") for the default reply-to address used by your Solarwinds installation. This should be a dedicated account that isn't used by any other applications or users.
  • A Solarwinds individual account ("the Solarwinds account") that can acknowledge alerts. This account must have the following permissions in Solarwinds:
    • Allow alert management rights = yes
    • Allow account to disable actions = yes
    • Allow account to disable alerts = yes
    • Allow account to disable all actions = yes
    • Allow Account to Clear Events, Acknowledge Alerts and Syslogs = yes
  • MS Exchange Web Services Managed API installed on the server that will run the script. Download the script from the MS Download Center.

Configuration:

  1. Log in to Windows as the account that will run the PowerShell script. Create a Windows credentials file for the email account and for the Solarwinds account. For each account, run the PowerShell command
    Get-Credential | Export-Clixml -Path [full path to file]
  2. In your Solarwinds alerts, configure the alert to send email. Make sure that the email body includes this text and variable:
    [AlertObjectID=${N=Alerting;M=AlertObjectID}]
  3. Edit the script as follows.
    1. Edit the line
      $exchangeEmail    = "solarwinds@mydomain.com"
      and replace "solarwinds@mydomain.com" with the address of the email account.

    2. Edit the line
      $SWServer = "solarwinds.mydomain.com"
      and replace "solarwinds.mydomain.com" with the fully-qualified domain name or IP address of your Solarwinds server

    3. If you are using a credentials file for the Exchange user, edit the line
      $ex_credstore = "c:\Users\solarwinds\Exchange_credentials.xml"
      and replace "c:\Users\solarwinds\Exchange_credentials.xml" with the full path to the Exchange credentials file you created.
    4. If you are using a clear text username and password for the Exchange user, edit the line
      $ex_credstore = "c:\Users\solarwinds\Exchange_credentials.xml"
      and replace "c:\Users\solarwinds\Exchange_credentials.xml" with "", then update the values for $ex_username and $ex_password on the next two lines.

    5. If you are using a credentials file for the Solarwinds user,
      $sw_credstore = "c:\Users\solarwinds\Solarwinds_credentials.xml"
      and replace "c:\Users\solarwinds\Solarwinds_credentials.xml" with the full path to the Solarwinds credentials file you created
    6. If you are using a clear text username and password for the Solarwinds user, edit the line
      $sw_credstore = "c:\Users\solarwinds\Solarwinds_credentials.xml"
      and replace "c:\Users\solarwinds\Solarwinds_credentials.xml" with "", then update the values for $sw_username and $sw_password on the next two lines.
  4. Copy the script to your Solarwinds server. You can run the script from the Powershell IDE to test it.
  5. Configure a task manager job to run the script at the desired interval. I run it once per minute to ensure that acknowledgement/comment emails are processed quickly enough to satisfy my end users.

Use:

This script assumes that the first word in the email message is a command verb such as ack or comment. Any text between the command and a return and/or newline character(s) is treated as a comment, and it will be appended to the alert in Solarwinds.

Message syntax:

At minimum, the email message must contain the Alert Object ID, formatted as described below. This element can appear anywhere in the message body.

Alert Object ID: A string of digits, in brackets, formatted like this:

    [AlertObjectID=99999]

For backward compatibility, the message may contain these elements instead of the Alert Object ID.

Alert definition ID: A GUID in brackets, formatted like this:</>

    [AlertDefinition=D1A5279D-B27D-4CD4-A05E-EFDD53D08E8D]

Object ID: A string of digits, in brackets, formatted like this:

    [ObjectID=99999]

Object Type: A string representing a valid Solarwinds object type, formatted like this:

    [ObjectType=APM: Application]

Commands:

The command must be the first word in the message body.

ack:    Acknowledge the alert and append a comment if one is supplied.

comment: Append a comment to the alert

Comments:

Comments are optional. Any text between the command verb and a newline or return character will be treated as a comment. There are no formatting requirements for comments.

Example 1:

This message would acknowledge an alert and append a comment

    Ack  Alert comment from ME

    [AlertObjectID=46673]

Example 2:

This message would also acknowledge an alert and append a comment.

    Ack Kurt is looking into this

    ~~~~~~~~~~~~~~~~~~~~~

    Me, Senior Systems Administrator

    ~~~~~~~~~~~~~~~~~~~~~~

    From: Solarwinds

    Sent: Friday, October 02, 2015 12:33 PM

    To: Me <me@myemail.com>

    Subject: Solarwinds Alert: Oracle database disk utilization KURTSDB on Node kurtsserver.mydomain.com is Down

    When replying, type your text above this line

    ________________________________________

    To acknowledge this alert, click here or reply to this message with "ack" as the first word in the body.

    Alert details:

    Oracle database disk utilization KURTSDB on Node kurtsserver.mydomain.com is Down at Friday, October 2, 2015 12:32 PM.

    When replying, do not delete text below

    ________________________________________

    [AlertObjectID=46673]

Example 3:

Deprecated format for backwards compatibility. This message would also acknowledge an alert and append a comment

    Ack  Third alert comment from ME

    [AlertDefinition=535d1493-a543-4df0-acbf-6b43770aceeb] [ObjectID=673] [ObjectType=APM: Application]

Attachments
Comments

Ah, very smart - rather than fighting the email format and grep'ing for the strings you need to do the work you force the creator of the email alert action to add some easily-identifiable markers. That's more than reasonable and saves you/us a lot of heartache.

This is brilliant and allows us to replace Alert Central which is absolutely horrible. Thank you for sharing.

Thanks! If you make any improvements or additions, please update in the comments here!

How does this replace event alert aggregation that Alert Central does?

It doesn't address Alert Central's alert aggregation directly. If you've configured Solarwinds to work with Alert Central, alert aggregation should work exactly as if you acknowledged the alert through the Solarwinds GUI.

Interesting solution, thanks for sharing. I just wanted to chip-in and share basics - just for the record - some of you might find it useful as well:

(1)

In the alert body use this:

<a href="https://SolarWindsServer-or-IP/Orion/NetPerfMon/AckAlert.aspx?AlertDefID=${SQL:SELECT AlertObjectID FROM AlertStatusView WHERE '${N=Alerting;M=AcknowledgeUrl}' LIKE '%='+CONVERT(NVARCHAR(MAX),AlertObjectID)}&viaEmail=True"><i>Acknowledge this alert and stop repetitive emails</i></a>

(2)

Result:

pastedImage_5.png

pastedImage_6.png

P.S. The reason I am not using "${N=Alerting;M=AcknowledgeUrl}" but instead re-create my own URL is because unfortunately this URL variable consist of FULL URL, which includes server name, whereas very often I just need to access server by IP and over https by default. So, that's why I am re-building the whole URL the way it suits me

That's it - pretty basic but hope will help you to look at it from another perspective as well

Obviously "comments" are not in this package

Very cool script, ran into a minor issue, when we ack the email it adds the entire email chain as the alert comments.

Nice. You should mention that you need to install the Exchange Web API as a Pre Req

This requires the Exchange Web Services Managed API to be installed on the computer where this script is being ran

# Download at - http://www.microsoft.com/en-us/download/confirmation.aspx?id=42022

This script will to run correctly and "processes" the email without error.  The Alert def object ID and Object type are being picked up by the script  However the alert is not being marked as "acknowledged" and remains unacknowledged.   I can not work out why.

We are on Exchange 2013 and NPM version 12

Good catch. I've added that to the prerequisites.

Get the XML payload created by this portion of the script:

    $xmlString = "<ArrayOfAlertInfo xmlns='http://schemas.solarwinds.com/2008/Orion'>

<AlertInfo>

    <DefinitionId>$($alertDefId)</DefinitionId>

        <ObjectType>$($objectType)</ObjectType>

        <ObjectId>$($objectId)</ObjectId>

    </AlertInfo>

</ArrayOfAlertInfo>"

How does the XML look?

How about the Solarwinds ID you're using to run the script - can you log in to Solarwinds with it and acknowledge alerts manually?

Hi M

Yes I can definitely login using the creds I provide to the script and it has full privileges.  I should get a chance to look at this tomorrow and let you know. about the XML payload   

I did have to amend the Exchange version as we are on 2013.  But as I said the script completes, writes the id's to host and  and sends a confirmation email so the issue is with the ack command not being actioned.  I will reply soon  thanks for checking in

Hi All, I have been looking for a solution to ack Alerts by email without using AlertCentral, looks like this is exactly what we need, checking on the requirements I see that script assumes we have Exchange Platform, so you know if works on the same way with exchange online? we are already moved to the cloud so just wondering to know if should work before doing all the config and testing for the script.

Thank you!

Does Exchange online have a dedicated URL associated with it? As long as this bit of the script works, you should be OK:

$s = New-Object Microsoft.Exchange.WebServices.Data.ExchangeService([Microsoft.Exchange.WebServices.Data.ExchangeVersion]::Exchange2007_SP1)

$s.Credentials = New-Object Net.NetworkCredential($exchangeUsername, $exchangePassword, $exchangeDomain)

$s.AutodiscoverUrl($exchangeEmail)

If autodiscover doesn't work with Exchange online, you should be able to set the Url property manually:

$s.Url = (your Exchange online URL)

hey, thanks for your prompt response, discovery is working on our environment so I don't see a major issue, I will try to get this done today.

Regards,

I am having an issue getting this to work. Autodiscovery is not working within powershell/ews for me (possibly due to our hybrid Office 365 environment). I have made the below change to attempt to resolve this:

# Discover the URL from the Exchange email address

#$s.AutodiscoverUrl($exchangeEmail)

$s.Url = "https://outlook.office365.com/EWS/Exchange.asmx"

However I am still getting the below error.

Exception calling "Bind" with "2" argument(s): "The request failed. The underlying connection was closed: An unexpected error occurred on a send."

At C:\scripts\process_alert_reply_email.ps1:286 char:1

+ $inbox = [Microsoft.Exchange.WebServices.Data.Folder]::Bind($s,[Microsoft.Exchan ...

+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

    + CategoryInfo          : NotSpecified: (:) [], MethodInvocationException

    + FullyQualifiedErrorId : ServiceRequestException

I have been trying for over a month to get this working, but I don't know powershell not to mention EWS. Any assistance would be greatly appreciated.

I haven't run into this error before, but it looks like this is usually a problem with SSL. One workaround is here: EWS Api problem connecting Inbox (look for Glen's reply about halfway down the page).

Is anyone monitoring this thread anymore? I was wondering if this would work with an O365 deployment?

I'm monitoring it, but we aren't using O365 so I can't answer your question.

jkrenzien​ , did you get this to work with your Office 365 environment?

We are doing an Office 365 migration and I was able to get it to work with changing the following line for autodiscover to the generic 365 address with no issues. This is generic for all 365 users so this should work

# Discover the URL from the Exchange email address\

$s.url = "https://outlook.office365.com/EWS/Exchange.asmx";

Also not a huge deal but you may want to change the code to point to the new version of MS Exchange Web Services Managed API which is 2.2 in the script. Easy update to do but wanted to point it out. Great job on this.

Thanks I will give that a try. Appreciate the response.

I have output that looks the same as yours and no acknowledgement. Any help would be appreciated. Looking at the DB table for AlertStatus there is nothing in any of the fields.

Can anyone do me a huge favor and send me the variables they are using in their alert setup.

I found the issue I am having and I believe EBeach is having and I am out of options on what variable we are using for ObjectID. Here is what I found.

For some reason none of the variables work for the ObjectID. They are IDS and the script processes them out because they meet the format but they come thru as unknown to SAM. I found by browsing the database SolarwindsOrion.AlertStatusView that if I take an email and switch the id to the one in the "activeObject" column it works like a champ. Issue is I can't find what variable that correlates to. If we can figure that out we should be good to go.

I would also put the variables in the script as a side note so its just an easy copy and paste to the bottom of your alert setup.

pastedImage_1.png

So here is the fix for it processing but not Acknowledging. I am sure others already have this but no one is sharing so I had to figure it out and put it together from scratch. I ended up using a SQL select statement and pulling the required info out for what the script needs. I went thru all alert variables and they do not have one for ActiveObject

Copy and paste this in the bottom of your alert and it should now pull what you need. If all agree this is a good fix, I think the owner should update the script and the setup instructions with this so its easier for users to setup.

Reply to this post if this helped so it can be marked as a fix.

Code to add to alert definition

When replying, do not delete text below

________________________________________

[AlertDefinition=${N=Alerting;M=AlertDefID}] [ObjectID=${SQL: SELECT max(ActiveObject) FROM AlertStatusView WHERE AlertObjectID like '${N=Alerting;M=AlertObjectID}'}] [ObjectType=${N=Alerting;M=ObjectType}]

It worked for me thanks for the help.

Thanks for coming up with this fix. What version of SAM and Orion are you using? Acknowledgements appear to be working properly for my installation using the code I posted. I just want to be sure this isn't related to the version of SAM or Orion before updating the instructions for all versions. Thanks.

Here is my version info.

pastedImage_0.png

Can you tell me what you have for your variables in the alert setup. The issue we have is that your statement requires the ObjectID which is 'ActiveObject' in the alert database table. I tried every variable known to the Solarwinds Gods and none of them would produce this number. Hence the SQL statement to pull it.

Understand this is not in the script. This is in the trigger section of setting up an alert. This is the section where SAM pulls the identifiers when the alert is emailed out. I was only stating to add this to the script in the comment section because there is no mention of which variables you would need to use for the identifiers when setting up your specific alert.

Example

pastedImage_0.png

The Active Object ID is different depending on the kind of object. For nodes, I just use the NodeID, like this:

pastedImage_1.png

For applications, I use the ApplicationID:

pastedImage_2.png

For disk volumes, I use the VolumeID:

pastedImage_3.png

I like your approach better, though - it's more universal, so it can be cut & pasted without regard for the type of object in the alert. I'll put a note in the script and publish the update.

Had to add the URL statement to the script to get it to run.  But now I am getting the following error when I try to run it from the server:

At C:\Users\jarrieh\Desktop\process_alert_reply_email.ps1:297 char:13

+ Write-Host "Number of unread messages in",$inbox.DisplayName,":",$inbox.UnreadCo ...

+             ~~~~~~

Unexpected token 'Number' in expression or statement.

At C:\Users\jarrieh\Desktop\process_alert_reply_email.ps1:312 char:88

+ ... ect" ]. Error: "$_.Exception.Message

+                    ~~~~~~~~~~~~~~~~~~~~~

The string is missing the terminator: ".

    + CategoryInfo          : ParserError: (:) [], ParseException

    + FullyQualifiedErrorId : UnexpectedToken

Can you paste line 297 from your code here? I'm not having problems with it on my system.

You can also try replacing it with this line:

Write-Host "Number of unread messages in",$($inbox.DisplayName),":",$($inbox.UnreadCount)

I figured it out, it was an extra " from the url line.  Now that is resolved, and it's spitting out new errors. I am going back through the comments chain now to see if someone has already come across and resolved the new errors.

I am trying to get this script working but I am currently running into a few errors.

$inbox = [Microsoft.Exchange.WebServices.Data.Folder]::Bind($s,[Microsoft.Exchange.WebServices.Data.WellKnownFolderName]::Inbox)

Exception calling "Bind" with "2" argument(s): "The response received from the service

didn't contain valid XML."

At line:2 char:1

+ $inbox = [Microsoft.Exchange.WebServices.Data.Folder]::Bind($s,[Microsoft.Exchan ...

+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

    + CategoryInfo          : NotSpecified: (:) [], MethodInvocationException

    + FullyQualifiedErrorId : ServiceRequestException

I am not really sure what to do next to try and troubleshoot the problem.

I had a few other issues when trying to run the script as well. See below just incase anyone looking over this runs into the issue.

When trying to run the same command:

$inbox = [Microsoft.Exchange.WebServices.Data.Folder]::Bind($s,[Microsoft.Exchange.WebServices.Data.WellKnownFolderName]::Inbox)

I would get the following error:

Exception calling "Bind" with "2" argument(s): "The request failed. The underlying connection was closed: Could not

establish trust relationship for the SSL/TLS secure channel."

This was resolved by making sure that I had the correct ssl certificate common name for the email server.

Can you dump the contents of $inbox? Does it look it ought to be valid XML, or is it obviously not?

I have deleted all of the emails in the inbox. When I had a test reply in the inbox I got the same error.  Does that error mean that the "ack" I responded to the alert with is incorrect?

Sorry, I meant can you show me the contents of the variable that is called $inbox in the script when you run it. I wonder if the script isn't getting a valid response from your email server.

"ack" is the correct response to the Solarwinds email. Make sure your reply email includes all of the stuff in brackets [] at the bottom of the message.

Hi Guys, I'm having problems on the execution of the script with the O365 configuration. I´ve changed the $s.Url parameter to $s.url = "https://outlook.office365.com/EWS/Exchange.asmx

and testing using a test account that is already on O365 for testing proposes, before migrate the actual account we're using for alerts, so I've also change the following parameters in the script with the information of this test account.

# Email address of the Exchange mailbox to read

$exchangeEmail    =

# Exchange user credentials, if not using your own

$exchangeUsername = "

$exchangePassword =

$exchangeDomain   =

When I executed the script it returns the following errors regarding unauthorized problem:

Exception calling "Bind" with "2" argument(s): "The request failed. The remote server returned an error: (401) Unauthorized."

At C:\Scripts\process_alert_reply_email - O365.ps1:295 char:1

+ $inbox = [Microsoft.Exchange.WebServices.Data.Folder]::Bind($s,[Microsoft.Exchan ...

+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

    + CategoryInfo          : NotSpecified: (:) [], MethodInvocationException

    + FullyQualifiedErrorId : ServiceRequestException

Exception calling "Bind" with "2" argument(s): "The request failed. The remote server returned an error: (401) Unauthorized."

At C:\Scripts\process_alert_reply_email - O365.ps1:298 char:1

+ $trash = [Microsoft.Exchange.WebServices.Data.Folder]::Bind($s,[Microsoft.Exchan ...

+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

    + CategoryInfo          : NotSpecified: (:) [], MethodInvocationException

    + FullyQualifiedErrorId : ServiceRequestException

You cannot call a method on a null-valued expression.

At C:\Scripts\process_alert_reply_email - O365.ps1:304 char:1

+ $items = $inbox.FindItems($batchSize)

+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

    + CategoryInfo          : InvalidOperation: (:) [], RuntimeException

    + FullyQualifiedErrorId : InvokeMethodOnNull

Does anyone can help me with this?. Is there any further configuration needed on the account from the Office365 team to be accessed by PS script? Or anything else I should change on the script?

Thanks

I got similar errors when they changed the load balancer configuration for our email server pool. I had to change the URL to point to a specific server in the pool. Are you also dealing with a load balancer? Can you try changing the URL to point to a specific server?

Did you ever find a fix for this?  I agree it is minor, but would be nice to have it fixed.  Also, is there a way to display ack comments on the alert page?  Thanks.

I haven't come up with a fix for this yet. I could probably do something where the email includes a warning to "comment above this line" and then truncate anything below the line. There is an "Alert notes" widget that I added to the Alert Details page in my SW implementation, and that's where the ack comments show up. Does your page have the "Alert notes" widget?    

I don't have the alert notes widget. Do you have a link please?

What version of SAM are you using? If you're using the newest version, you should be able to search for the Alert Notes widget when you click the Customize Page icon, like this:

Alert_Details_Add_Widget.png

If you're using an older version, you can search for the Alert Notes resource. Click the Customize Page icon and then click the Add resource icon:

Active_Alert_Details_alert_notes.png

Does that work?

Hey Alex,

Do you by chance have it so that it prompts you for a note before had?

Just we ask our internal teams to log it in our tracking system and put the Incident details there?

Thanks

Awesome post. Thank you!

Hey Milligan.... I have been thinking for a while now that you should include an "Unmanage" option in the script. This way if the node was supposed to be unmanaged and you forgot you can simply reply with Unman and add comments if you want and it will go thru and acknowledge the alert (and add comments if they are there) and then it will unmanage the node for a set time. With that in mind I went ahead and added the functionality to your script. For the unmanage I only have it set to unmanage for 30 minutes and then it will remanage. PM me if you want it.

That's a cool idea. I can look at working that into the next version. Maybe require "unman" followed by the number of minutes to unmanage (or indefinitely if there's no number)?

Code is all done. I will see if I can PM it to you to vet and then you can update it on your page if you are happy with it. Unman will just unman for 24 hours (you can change it to whatever you like, I have people who won't remanage it so I set it for 24 lol) and if you do unman (time in minutes) so unman 15 it will unmanage the node for 15 minutes. Next version I will look at adding a mute option and also maybe a group unman option. Just have to find the time to put it together. Again you laid the ground work so this is your baby. I am just adding to your madness!!

Firstly, thanks for the script, it works well in our environment. I just have a question regarding the acknowledgment process, which I would be grateful if someone could help with.

When an alert is triggered, it appears within active alerts but when the email acknowledgment is received, it seems that the alert is reset because it disappears out of active alerts. We were hoping that the alerts would remain active and just show as acknowledged with a comment.

Has anyone else experienced this? Is this a feature of the script or NPM 12.3 ?

The script is only acknowledging the alert is is not setup to clear it. If it is clearing it, it is by some other means. I would start by looking on the alert - If you edit the alert and take a look at reset conditions - This will clear it depending on how you have it set. I would recommend testing this as some alerts will not refire if they are not reset from my experience. Even if you choose "No reset condition" Reset Condition When the reset condition is met the alert is removed from active alerts.

»Learn more Reset this alert automatically after 6 Minutes    No reset condition – Trigger this alert each time the trigger condition is met No reset action - Manually remove the alert from the active alerts list Create a special reset condition for this alert

The script only acknowledges the alert. It doesn't clear it. I second adbs98 - check your alert reset configurations to see if they're set to clear themselves after a few minutes.

We don't have anything that would reset the alert in the conditions, so its a bit strange. 

Version history
Revision #:
1 of 1
Last update:
‎01-14-2016 05:08 PM
Updated by: