14 Replies Latest reply on Jan 6, 2010 1:31 PM by Steven Klassen

    How do you get alerts to your techs?

    byrona

      I am curious what unique alerting schemes other people and companies out there are using with Orion to get alerts to their techs?

      My company wants me to automate things as much as possible and minimize the need for a person to be watching the monitoring system dashboards to identify a problem.  The only practical way I have to alert people is via email which has some downfalls as well.

        • Re: How do you get alerts to your techs?
          Steven Klassen

          There's the email route, and then there's the SMS option using the PageGate integration. It involves picking up another piece of software, however:

          http://www.notepage.net/

          • Re: How do you get alerts to your techs?
            DirtySouth

            We use email alerts for pretty much everything. Depending on the importance of the device, we alert at either 1 minute, 5 minute or 10 minute downtime intervals. Were you interested in the actual verbiage itself in the alerts? If so, I can post some examples. If you're concerned about email being unavailable, SMS is a good option.

              • Re: How do you get alerts to your techs?
                byrona

                Thanks for the feedback thus far.

                I am no specifically interested in the contents of the alert, just curious what methods people are using to get the alerts to the techs.

                SMS is a good idea and most cell phone companies have an email to text gateway you can use as well.

                With the NMS system that we are replacing with Orion we use XMPP as we have an internal chat server unfortunately with Orion that is not an option.  I did however submit that as a feature request.

                  • Re: How do you get alerts to your techs?
                    Steven Klassen

                    Why can't you use XMPP with Orion? The alert actions have a "run external program" option that you could easily point to a script written in any number of languages using an XMPP library to pass along a message to someone. A quick google search found examples for perl (Net::XMPP), and python (xmpp library) to name a few.

                      • Re: How do you get alerts to your techs?
                        byrona

                        I was referring to the fact that XMPP is not an out-of-the-box option with Orion as it is with my current system that I am replacing.

                        I realize that this is possible using a scripted solution.  Through the use of scripts and external programs one could feasibly argue that anything is possible.

                          • Re: How do you get alerts to your techs?
                            Steven Klassen

                            There would be an hour's worth of development involved if you got a hold of someone with some scripting savvy, but XMPP definitely doesn't fall under "not an option."

                              • Re: How do you get alerts to your techs?
                                byrona

                                Yeah, I took a look at those libraries you noted and the Python one would probably work out well here, thanks for pointing those out.  We do have script savvy folks and even a full on developer and I may point them to that if the decisions is we need to maintain that XMPP functionality.

                                That being said, if it's only a few hours of work, how awesome would it be to have it as an out-of-the-box option.

                                Our current monitoring solution is very heavily home grown with a LOT of scripted and hand written components and we want to move away from that as much as possible which is one of the reasons we choose Orion.

                                Thanks again for the suggestions!

                                  • Re: How do you get alerts to your techs?
                                    ErikInHell

                                    I just wanted to pose a question on this thread, as it might be something for me to add once I get my Solarwinds up and running.  Is there an "either/or" function on alerts, so if I'm using email alerts and the email server has a failure, I can use the paging server we already have in place?

                                    I don't want to send double alerts by using an email and paging service, and it would be nice to know if I can set that up in the alerting function, or if I'd have to set it up separately as a service elsewhere.

                                    Thanks.

                                      • Re: How do you get alerts to your techs?
                                        Steven Klassen

                                        No, the alert actions are best-effort. If you test your alert action you can tell whether or not it was successful, but left on its own you could be sending email into the ether for weeks and never know about it unless you realized things were a bit too quiet.

                                        I'll have to give that some thought. My initial thinking is an escalation alert action that only goes off if the alert hasn't been acknowledged after X minutes (meaning either the person that was notified is sleeping on the job or they never got the email).

                                          • Re: How do you get alerts to your techs?
                                            ErikInHell

                                            It would be more like sleeping on call than on the job, but that might be a good way to do it.  If I set up the escalation alert for only the critical devices, it would assure that someone gets a page for something major.

                                              • Re: How do you get alerts to your techs?
                                                Steven Klassen

                                                Good call. Make sure you're using a custom property to identify the critical devices. For that matter, make sure you're using one for the email address. Not many folks know that you can drop ${Node.Contact_Email} in the To: field to make it dynamic.

                                                  • Re: How do you get alerts to your techs?
                                                    robertjuric


                                                    Good call. Make sure you're using a custom property to identify the critical devices. For that matter, make sure you're using one for the email address. Not many folks know that you can drop ${Node.Contact_Email} in the To: field to make it dynamic.

                                                     



                                                    Where do you set the ${Node.Contact_Email} property?

                                                     

                                                    For our field equipment we have separate alerts that send e-mails to the corresponding field tech of a device. The way it is currently setup I have an alert for each field tech, but I may look into changing the setup to something like mrxinu suggested to make a single dynamic alert. The alerts also e-mail our Service Desk group to start a ticket on the trouble

                                                    For our core equipment we have an alert with time-of-day based action. During business hours the alert sends an e-mail to our Service Desk and Infrastructure groups. After hours we send alerts to an answering service who calls our on-call technician and informs them of the alert.

                                                    We found that e-mails alone were not sufficient or reliable enough to wake up our on-call technicians at 2am. Our  answering service is invaluable to us.

                                            • Re: How do you get alerts to your techs?
                                              Steven Klassen

                                              It would be cool, but having been a UI developer in a previous life there's a big difference between kicking out a script that handles a few command line options (recipient, node name, IP address, etc), opening a connection to the XMPP server, and sending the message; and building up a UI to accept all that information, validate it, and then do the first bit under the covers.