Automated alert acknowledgement from remote script

rmarshall

I would like to be able to acknowledge alerts in Orion from a script on another host.  It would be ideal if I can accomplish this with some sort of Python script.  I tried doing an http post from Python, but I guess since it is not a browser it is not handling the redirect as a single session and just redirects to login.asp over and over. 

Has anyone solved this problem before? 

Is there a way to acknowledge an alert with an HTTP get or post while providing credentials that won't result in a redirect?

Is there any other way to programmatically acknowledge an alert using a shell script or Python script?

I am using SolarWinds Orion Network Performance Monitor 9.0 SP2.

rmarshall

OK, I have successfully completed my Solarwinds Orion monitoring enhancements so that the following have been accomplished:

• If a core switch fails, instead of getting hundreds of Node Down alerts we get one email with 100 alerts in it. 
• We can acknowledge all of the alerts by replying to the email with "ack" in the message body
• People can email the monitoring system to subscribe or unsubscribe to Solarwinds alerts.
• Email alerts for Node Down get sent with Important High which generate a Level 1 alert and thus an audible alarm on the team's Blackberries
• After hours the first node down alert (Level 1 and thus audible) goes to the oncall person and if it is not acknowledge in 15 minute the alert is sent to the entire DC Ops team and if no one acknowledged the alert in another 15 minutes the managers are added onto the alert.
• The oncall person rotates every Monday at 8:00 automatically.
• A summary email is generated each hour without the Importance High/Level 1 switch to the entire DC Ops team including the managers which shows all alerts that were generated that hour.

The way I accomplished this was to send all Solarwinds alerts to Splunk as syslog messages then run a python script on the Splunk host every five minutes to check for new Solarwinds alerts.  Three python scripts and an IMAP pluggin to Spunk for grabbing the emails provide all of the functionality. 

Splunk is a log and message data indexing tool that has proven quite handy to us.  They have a free download that if all I was doing was using it to manage my Solarwinds alerts I would not generate enough traffic to exceed the free license.  You can download splunk at http://www.splunk.com.  The IMAP email reader is a free download/plugin found from inside the product.

I am using version 3.4.6 of Splunk and version 9.0 of Orion.  You will have to modify the code below appropriately with your own values.  I just wrote it and I haven't cleaned it up, so excuse the messiness of my code, lack of a settings section and lack of comments.

 

The files that make up this solution on the splunk host are as follows:

/root/alertmon.py   - the main alerting script run every 5 minutes

/root/ack.py   - script that acknowledges alerts on Solarwinds

/root/amonhourly.py    - generates hourly summary of active alerts

/root/amonsubscribers.txt    - list of subscribers to the Solarwinds alerts - users are added by emailing splunk

/root/amononcall.txt    - on-call schedule - one email address per line (no delimiters, just a list) on-call person identified by * before email address

/root/amonescalation.txt   - On-call escalation list (List of manager emails)

/etc/crontab   - Runs the files in cron.five and cron.hourly

/etc/cron.five/alertmon.sh    - Runs the alertmon script every five minutes

/etc/cron.hourly/dfemail.sh   - Runs the hourly summary script

Note: you have to set your environment variable inside the cron shell scripts in order to run the python scripts

 

Here are the contents of each python script:

 

alertmon.py

#!/opt/splunk/bin/python
import smtplib, csv, time, sys, re, time
from ack import ackAlert, extractcookies
import MimeWriter, StringIO, base64
from splunk import auth, search

#############  This section is for checking for subscibe and unsubscribe requests

auth.getSessionKey('splunkuserhere','splunkpwhere')

def numsort(x, y):
    if y[1] > x[1]:
        return -1
    elif x[1] > y[1]:
        return 1
    else:
        return 0

def pad(count):
    padding = ""
    for i in range(0, count):
        padding += ' '
    return padding

###  Check for ack messages and acknowledge alerts
def checkForAck():
    job = search.dispatch('search index::mail ack startminutesago=10')
    while not job.isDone:
       time.sleep(1)
    ack=0
    AlertGUID=""
    NodeID=""
    InterfaceID=""
    for event in job.events:
        #print event.fields['_raw']
        raw = event.fields['_raw'].split('\n')
        print "@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@"
        for row in raw:
            print row +"  "
            if '{' in row:
                AlertGUID=row[(row.index('{')+1):row.index('}')]
                print "AlertGUID="+AlertGUID
            if ('(' in row) and ('NodeID:' in row):
                NodeID=row[(row.index('(')+1):row.index(')')]
                NodeID=NodeID[NodeID.index(':')+2:]
                print "NodeID="+NodeID
            if ('(' in row) and ('InterfaceID:' in row):
                InterfaceID=row[(row.index('(')+1):row.index(')')]
                InterfaceID=InterfaceID[InterfaceID.index(':')+2:]
                print "InterfaceID="+InterfaceID
            if ack==1 and NodeID!="":
                print "\n\nACK ALERT: "+AlertGUID+" : "+NodeID+"\n\n"
                ackAlert(AlertGUID, NodeID, "Node")
                NodeID=""
            elif ack==1 and InterfaceID!="":
                print "\n\nACK ALERT: "+AlertGUID+" : "+InterfaceID+"\n\n"
                ackAlert(AlertGUID, InterfaceID, "Interface")
                InterfaceID=""

            if row[:3].lower()=="ack":
                print "\n\nACK FOUND!!!!!!!!!!!!\n\n"
                ack=1
    job.cancel()
    return


# Check if any alerts have occured in the last five minutes
def checkForAlerts():
    job = search.dispatch('search host=solarwinds monitoringalert Ack="Not Acknowledged" startminutesago=5 | dedup NodeID| sort "Severity", "AlertGUID", "NodeID"')
    while not job.isDone:
       time.sleep(1)
    print job.count
    bodytext=""
    events=0
    Severity=10
    Alert=""
    for event in job:
        print event
        events+=1
        # print event.fields['Alert']
        if Alert!=event.fields['Alert']:
            Alert=event.fields['Alert']
            bodytext+="\n\n***  "+event.fields['Alert']+"  ***\n{"+event.fields['AlertGUID']+"}\n\n"
        AlertGUID=event.fields['AlertGUID']
        print "Severity: "+event.fields['Severity']
        if int(event.fields['Severity'])<Severity: Severity=int(event.fields['Severity'])
        NodeID=event.fields['NodeID']
        Nodename=event.fields['Nodename']
        IP=event.fields['IP']
        Ack=event.fields['Ack']
        padsize=29-len(Nodename)
        bodytext+=Nodename+pad(40-len(Nodename))+IP
        if event.fields.has_key('InterfaceID'):
            InterfaceID=event.fields['InterfaceID']
            bodytext+=" (InterfaceID: "+InterfaceID+")\n\n"
        else:
            bodytext+=" (NodeID: "+NodeID+")\n\n"
    job.cancel()
    return bodytext, Severity

      
def checkForSubscriptions():
    ####  Check for new subscribers
    print "#####   CHECKING FOR SUBSCRIPTION EMAIL  ####"
    job = search.dispatch('search index::mail startminutesago=10 sub SM | reverse')
    while not job.isDone:
       time.sleep(1)
 
   # this will stream events back until the last event is reached

    for event in job:
        #print event
        subscriber =  event.fields['From'][event.fields['From'].index("<")+1:event.fields['From'].index(">")]
        newSubList=[]
        try:
            thresh=int(event.fields['Subject'][event.fields['Subject'].lower().index("sm")+3:])
            #print "Setting threshhold to: "+event.fields['Subject'][event.fields['Subject'].index("sm")+3:]
        except:
            #print "No integer found.  Defaulting to 0"
            thresh=0
        yesFlag=0       
        fin = open("amonsubscribers.txt","r")
        subscriberList = fin.readlines()
        fin.close()
        csvReader = csv.reader(subscriberList, delimiter=':')
        for row in csvReader:
            if row[0]==subscriber:
                print subscriber+" is in the list"
                if event.fields['Subject'][0:3].lower()=="sub":
                    print "Setting subsriber threshhold to: "+str(thresh)
                    a=subscriber+":"+str(thresh)+"\n"
                    newSubList.append(a)
                    yesFlag=1
                elif event.fields['Subject'][0:5].lower()=="unsub":
                    print "Removing: "+row[0]
            else:
                print "keeping: "+row[0]
                a=row[0]+":"+row[1]+"\n"
                newSubList.append(a)
        if yesFlag==0 and event.fields['Subject'][0:3].lower()=="sub":
            a=subscriber+":"+str(thresh)+"\n"
            newSubList.append(a)
            print "Adding: "+subscriber
        print newSubList
        fout = open("ddsubscribers.txt","w")
        fout.writelines(newSubList)
        fout.close()
        print "###################################"
        print "  "
    job.cancel()

def rotateOnCall():
    fin = open("amononcall.txt","r")
    subscriberList = fin.readlines()
    fin.close()
    i=0
    n=0
    print 'len: '+str(len(subscriberList))
    for row in subscriberList:
        if row[0:1]=="*":
            subscriberList=subscriberList[1:]
            n=i+1
        i=i+1
    if n>(len(subscriberList)-1):n=0
    print 'n: '+str(n)
    print subscriberList[n]
    print '*'+subscriberList[n]
    subscriberList[n]='*'+subscriberList[n]
    fout = open("amononcall.txt","w")
    fout.writelines(subscriberList)
    fout.close()
 

bodytext, Severity=checkForAlerts()
checkForAck()
checkForSubscriptions()
if time.strftime('%A')=='Monday' and time.strftime('%H %M')=='08 00':rotateOnCall()
print bodytext
#sys.exit("Program ended.")

##############  This section starts the process of building and sending the email to the subscriber list
print "#####################################Top of Output########################################\n"

# Create the message
serverURL  = "SMTPServerIPHere"                     #argvals.get("server", "localhost")
sender     = "splunk@yourdomain.com"              #argvals.get("from", "splunk@" + socket.gethostname())
subject = "Severity "+str(Severity)+" Monitoring Alert"
importance = "3"                                #argvals.get("priority", None)


message = StringIO.StringIO()
writer = MimeWriter.MimeWriter(message)
writer.addheader('Subject', subject)


importanceMap = {
    "highest": "1 (Highest)",
    "high"   : "2 (High)",
    "low" : "4 (Low)",
    "lowest" : "5 (Lowest)",
    "1" : "1 (Highest)",
    "2" : "2 (High)",
    "4" : "4 (Low)",
    "5" : "5 (Lowest)"
}

recipients = []
if int(time.strftime('%H'))>8 and int(time.strftime('%H'))<18 and time.strftime('%A')!='Saturday' and time.strftime('%A')!='Sunday':
    fin = open("amonsubscribers.txt","r")
    subscriberList = fin.readlines()
    fin.close()
    csvReader = csv.reader(subscriberList, delimiter=':')
    for row in csvReader:
        if Severity<=int(row[1]) or int(row[1])==0:
            writer.addheader('To', row[0])
            recipients.append(row[0])
            print "Sending alert to: "+row[0]
        else:
            print "Not sending alert to: "+row[0]
            print str(Severity) +" and "+row[1]
elif bodytext!='':
    if Severity==1:
        job = search.dispatch('search index::mail startminutesago=17 endminutesago=12 escalation_level')
        while not job.isDone:
           time.sleep(1)
        
        # this will stream events back until the last event is reached
        print 'job count: '+str(job.count)
        if job.count>0 and Severity==1:
            print "escalation: "+job.events[0].fields['escalation_level']
            if job.events[0].fields['escalation_level']=='3D3':     # This should just be a 3, but a glitch in the IMAP reader currently adds 3D to the front
                bodytext=bodytext+"\n\nescalation_level=2\n"
            else:
                bodytext=bodytext+"\n\nescalation_level=1\n"
                fin = open("amonescalation.txt","r")
                subscriberList = fin.readlines()
                fin.close()
                for row in subscriberList:
                    writer.addheader('To', row)
                    recipients.append(row)
                    print "Sending alert to: "+row

            fin = open("amonsubscribers.txt","r")
            subscriberList = fin.readlines()
            fin.close()
            csvReader = csv.reader(subscriberList, delimiter=':')
            for row in csvReader:
                if Severity<=int(row[1]) or int(row[1])==0:
                    writer.addheader('To', row[0])
                    recipients.append(row[0])
                    print "Sending alert to: "+row[0]
                else:
                    print "Not sending alert to: "+row[0]
                    print str(Severity) +" and "+row[1]
            writer.addheader('To', 'splunk@mydomain.com')
            recipients.append('splunk@mydomain.com')

        else:
            bodytext=bodytext+"\n\nescalation_level=3\n"
            writer.addheader('To', 'splunk@mydomain.com')
            recipients.append('splunk@mydomain.com')

        job.cancel()
    fin = open("amononcall.txt","r")
    subscriberList = fin.readlines()
    fin.close()
    for row in subscriberList:
        if row[0:1]=='*':
            writer.addheader('To', row[1:])
            recipients.append(row[1:])
            print "Sending alert to: "+row[1:]
  

if Severity < 3: importance=str(Severity)
if importance != None:
    # look up better name
    val = importanceMap.get(importance.lower(), None)
    # unknown value, use value user supplied
    if val == None:
       val = importance
    writer.addheader('X-Priority', val)


writer.startmultipartbody('mixed')

bodyformat = "text/plain"


#######################################################################################
# create the body of the email and attach or inline results if required. Make sure to #
# adhere to the requested format and proper tag balancing.                            #
#######################################################################################
if bodytext != "" and recipients!=[]:
    bodytext = bodytext+"\n\nThis is a monitoring alert. To acknowledge this alert reply with 'ack' in the body."
    part = writer.nextpart()
    body = part.startbody(bodyformat, [("charset", 'us-ascii')])

    #body.write("<HTML><BODY>\n")
    body.write(bodytext)
    #body.write("</BODY></HTML>")   

    # finish off email writting/composition
    writer.lastpart()


    server = smtplib.SMTP('SMTPServerIPHere')
    server.set_debuglevel(True) # show communication with the server
    try:
        server.sendmail(sender, recipients, message.getvalue())
    finally:
        server.quit()

 

 

ack.py

#!/usr/bin/env python

import sys
import httplib

def extractcookies(response):
        cookies = ''

    allcookies = response.getheader('Set-Cookie')
        if allcookies != None and len(allcookies) > 0:
        if allcookies.find(', '):
                cookie = allcookies.split(', ')
                for c in cookie:
                        if len(cookies) == 0:
                                cookies = c[0:c.find(';')]
                        else:
                                cookies = cookies + '; ' + c[0:c.find(';')]
        else:
            cookies = cookie[0:cookie.find(';')]

        print 'Cookies:<'+cookies+'>\n'
    return cookies;

def ackAlert(alertGUID, NodeID, Type="Node"):
    starturl = 'SolarwindsIPHere/.../Login.aspx + alertGUID + '%7%3d9%26ack%3d%7b' + alertGUID + '%7d%3a' + NodeID + '%3a'+Type
    # starturl = sys.argv[1] #For testing
    print '\nstarturl=' + starturl + '\n'

    location, querystring  = starturl.split('?', 1)
    host = location.lstrip('http://')
    host, uri = host.split('/', 1)
    uri = '/' + uri
    print 'host=<' + host + '>, uri=<' + uri + '>, querystring=<' + querystring + '>\n'

    conn = httplib.HTTPConnection(host)
    conn.request('GET', uri+'?'+querystring)
    response = conn.getresponse()
    print 'status='+str(response.status)+', reason='+response.reason+'\n'
    cookies = ''
    while response.status == 302:
    if response.getheader('Set-Cookie') != None:
        cookies = extractcookies(response)
    location = response.getheader('Location')
    conn.close
    print 'Following redirect to ' + location + '\n'
    conn = httplib.HTTPConnection(host)
    conn.putrequest('GET', location)
    if len(cookies) > 0:
        conn.putheader('Cookie', cookies)
    conn.endheaders()
    conn.send('')
    response = conn.getresponse()

    print 'final response: ' + str(response.status) + ', reason=' + response.reason
    conn.close

if len(sys.argv) == 3:
    alertGUID=sys.argv[1]
    nodeID= sys.argv[2]
    ackAlert(alertGUID, nodeID)
else:
    print 'usage: ' + sys.argv[0] + ' solarwinds-ack-url'
       

amonhourly.py

#!/opt/splunk/bin/python
import smtplib, csv, time, sys, re, time
from ack import ackAlert, extractcookies
import MimeWriter, StringIO, base64
from splunk import auth, search

##### This is a copy of the alert monitoring script modified to provide hourly summaries of all alerts without the Level 1 switch

#############  This section is for checking for subscibe and unsubscribe requests

auth.getSessionKey('splunkusername','splunkpwhere')

def numsort(x, y):
    if y[1] > x[1]:
        return -1
    elif x[1] > y[1]:
        return 1
    else:
        return 0

def pad(count):
    padding = ""
    for i in range(0, count):
        padding += ' '
    return padding


def checkForAlerts():
    job = search.dispatch('search host=solarwinds monitoringalert Ack="Not Acknowledged" startminutesago=60 | dedup NodeID| sort "Severity", "AlertGUID", "NodeID"')
    while not job.isDone:
       time.sleep(1)
    print job.count
    bodytext=""
    events=0
    Severity=10
    Alert=""
    for event in job:
        print event
        events+=1
        # print event.fields['Alert']
        if Alert!=event.fields['Alert']:
            Alert=event.fields['Alert']
            bodytext+="\n\n"+event.fields['Alert']+" \n\n"
        AlertGUID=event.fields['AlertGUID']
        print "Severity: "+event.fields['Severity']
        if int(event.fields['Severity'])<Severity: Severity=int(event.fields['Severity'])
        NodeID=event.fields['NodeID']
        Nodename=event.fields['Nodename']
        IP=event.fields['IP']
        Ack=event.fields['Ack']
        padsize=29-len(Nodename)
        bodytext+="  -"+Nodename
    job.cancel()
    return bodytext, Severity, events

def checkForAlertResets():
    job = search.dispatch('search host=solarwinds monitoringreset startminutesago=60 | dedup NodeID| sort "Severity", "AlertGUID", "NodeID"')
    while not job.isDone:
       time.sleep(1)
    print job.count
    bodytext=""
    events=0
    Alert=""
    for event in job:
        print event
        events+=1
        # print event.fields['Alert']
        if Alert!=event.fields['Alert']:
            Alert=event.fields['Alert']
            bodytext+="\n\n"+event.fields['Alert']+" - Alert Resets\n\n"
        Nodename=event.fields['Nodename']
        IP=event.fields['IP']
        Ack=event.fields['Ack']
        padsize=29-len(Nodename)
        bodytext+="RESET:"+Nodename
    job.cancel()
    if bodytext != "": bodytext="\n\nMonitoring Alert Resets in last hour:"+bodytext
    return bodytext

      
bodytext, Severity, events=checkForAlerts()
bodytext+=checkForAlertResets()
print bodytext
#sys.exit("Program ended.")

##############  This section starts the process of building and sending the email to the subscriber list
print "#####################################Top of Output########################################\n"

# Create the message
serverURL  = "MySMTPServerIP"                     #argvals.get("server", "localhost")
sender     = "splunk@mydomain.com"              #argvals.get("from", "splunk@" + socket.gethostname())
subject = "Solarwinds Alerts This Hour: " + str(events)
importance = "3"                                #argvals.get("priority", None)


message = StringIO.StringIO()
writer = MimeWriter.MimeWriter(message)
writer.addheader('Subject', subject)

importanceMap = {
    "highest": "1 (Highest)",
    "high"   : "2 (High)",
    "low" : "4 (Low)",
    "lowest" : "5 (Lowest)",
    "1" : "1 (Highest)",
    "2" : "2 (High)",
    "4" : "4 (Low)",
    "5" : "5 (Lowest)"
}

recipients = []
fin = open("amonsubscribers.txt","r")
subscriberList = fin.readlines()
fin.close()
csvReader = csv.reader(subscriberList, delimiter=':')
for row in csvReader:
    if Severity<=int(row[1]) or int(row[1])==0:
        writer.addheader('To', row[0])
        recipients.append(row[0])
        print "Sending alert to: "+row[0]
    else:
        print "Not sending alert to: "+row[0]
        print str(Severity) +" and "+row[1]
        job.cancel()
fin = open("amonescalation.txt","r")
subscriberList = fin.readlines()
fin.close()
for row in subscriberList:
    writer.addheader('To', row)
    recipients.append(row)
    print "Sending alert to: "+row
  

writer.startmultipartbody('mixed')

bodyformat = "text/plain"


#######################################################################################
# create the body of the email and attach or inline results if required. Make sure to #
# adhere to the requested format and proper tag balancing.                            #
#######################################################################################
if bodytext != "" and recipients!=[]:
    bodytext = "This is a summary of monitoring alerts that have occured in the last hour.  Do not reply to this message."+bodytext
    part = writer.nextpart()
    body = part.startbody(bodyformat, [("charset", 'us-ascii')])

    #body.write("<HTML><BODY>\n")
    body.write(bodytext)
    #body.write("</BODY></HTML>")   

    # finish off email writting/composition
    writer.lastpart()


    server = smtplib.SMTP(serverURL )
    server.set_debuglevel(True) # show communication with the server
    try:
        server.sendmail(sender, recipients, message.getvalue())
    finally:
        server.quit()

Solarwinds Configuration

Alerts are sent to splunk from solarwinds by configuring an Advanced Alert for Node Down and sending a syslog message to Splunk.

 

On Trigger Actions, the message body looks like this:

Severity=1 | AlertGUID=${AlertDefID} | NodeID=${NodeID} | Alert=${Alert} | Nodename=${NodeName} | IP=${Node.IP_Address} | Ack=${Acknowledged} | MonitoringAlert

 

On the Alert Escalation tab the checkbox is check to "Execute this alert repeatedly while the alert is triggered" with it set to repeat every 15 minutes.

 

The Reset Actions has a message body of:

RESET: AlertGUID=${AlertDefID} | NodeID=${NodeID} | Alert=${Alert} | Nodename=${NodeName} | Ack=${Acknowledged} | MonitoringReset

 

 

Alert Flow

When an alert is generated in Solarwinds, a syslog message is sent to Splunk with the alert data in the message body.  The Node Down alert is the primary alert sent through this process, but other alerts can be sent to splunk as well. 

 

Note: The code currently deduplicates any NodeID/Nodename found so this system may have to be changed if it is to handle multiple alerts and different kinds of alerts.

 

Every five minutes a cron job on the Splunk host runs the alertmon.py script which queries Splunk for "MonitoringAlert".  This is the search performed by the script to pick up new alerts:

search host=solarwinds monitoringalert Ack="Not Acknowledged" startminutesago=5 | dedup NodeID| sort "Severity", "AlertGUID", "NodeID"

 

During the hours of 8:00AM to 6:00PM M-F, if one or more alerts are found an email is sent to the subscribers list. 

 

After hours a message is sent to just the on-call person (amononcall.txt).  If it is not acknowledged within 15 minutes it goes to the entire subscribers list.  Managers are added if it is not acknowledged 15 minutes after that.   Escalation level is monitored by placing the "escalation_level=3,2 or 1" tag in the message body and including splunk in the "To" list.  Escalation level 1 is the highest level.

 

Every time the alertmon.py script runs (every 5 minutes) it also checks for emails sent to splunk.  It checks for escalation level for messages after hours and it checks for alert acknowledgements. 

 

If an email sent back to splunk has "ack" as the first three letters of any line of text in the email it then parses the message one line at a time looking for the alert GUID and then subsequent NodeID entries.  Using the functions in the ack.py script, Splunk sends and acknowledgement for each alert via HTTP request to Solarwinds (providing the Alert GUID and NodeID).

 

The last thing the script does is send out an email to the recipients list if any "Not Acknowledged" alerts have been found in the last five minutes.

 

Every hour a separate script called amonhourly.py sends an email to everyone in the subscribers (amonsubscribers.txt) and escalation (amonescalation.txt) lists with a summary of all alerts triggered and reset in the last hour.  This alert is not flagged as Level 1/Importance High.

 

Instructions to End Users

 

Users can subscribe or unsubscribe to the Solarwinds monitoring alerts by sending an emails to splunk@mydomain.com with the following in the Subject:

To subscribe:  "sub SM"

To unsubscribe:  "unsub SM"

 

You can acknowledge alerts by replying to splunk with ack (caps or lower case) in the body. 

 

After hours the oncall person gets the first message. Then subsequent messages go to everyone if it is not ack'd in 15 minutes.  After 30 minutes the managers are added to the alerts.  The on-call notification rotates to the next person on-call at 8:00AM every Monday.

 

You can setup your BB to beep on Severity 1 messages by doing two steps:

1. Setup your profile to alert for level 1 messages inside your BB

 

1. Using the BB Desktop Manager, create a filter for Important messages
   1. Click on Email Settings/Redirector Settings (depending on version of BB Desktop)
   2. Go to Filters tab
   3. Create a new filter
      1. Call the filter Important or Severity 1
      2. Check the Importance box and set it to High
      3. Check the box for Forwarding as Level 1
   1. Save your new filter

NOTE: Consider also filtering on emails from  splun@mydomain.com if you don’t want all emails set with Importance High to generate an alert.