26 Replies Latest reply on Nov 16, 2015 3:55 PM by johnny ringo

    Third Party Application Acknowledge SolarWinds Event

    cmwillis

      Hello,

       

      I am currently working to find a solution for acknowledging SolarWinds events from an IBM Tivoli Netcool server. I am already generating SNMP alerts using the SolarWinds Advanced Alert Manager. Netcool processes these alerts and they are displayed in a color coded dashboard. Using Tivoli Netcool I can click on an SNMP alert received from SolarWinds and run an executable to "do something." Currently I have the tool configured to start a command prompt and run the command:

       

      start "" "http://<SolarWindsServerHostname>/Orion/NetPerfMon/AckAlert.aspx?AlertDefID=@AlertKey:@NodeAlias:@AlertGroup"

       

      where:

      @AlertKey = SolarWinds ${PropertyValue}

      @NodeAlias = SolarWinds ${NodeID}

      @AlertGroup = SolarWinds ${NetObjectType}

       

      However, this causes the browser to be launched every time and the user is prompted to authenticate with either Windows logon credentials or a SmartCard. Ideally, I would like to hide the browser and only prompt the used for their credentials on the first authentication attempt (or somehow use cached credentials on the client). I havent been able to find anything that works like this. I created a PowerShell script that attempts to use System.Net.WebClient to accomplish the task, but that throws back a 401 error when authenticating with the SolarWinds server's IIS platform. I think this is attributed to the fact that I have Windows authentication required, but I am not sure. IIS doesn't appear to complain in its logs about anything.

       

      Does anyone know of a better way to do what I am trying to accomplish? I have looked into using the SolarWinds SDK, but it doesnt appear to provide anything to tweak the database or acknowledge events.

       

       

        • Re: Third Party Application Acknowledge SolarWinds Event
          Jan Pelousek

          Hello,

           

          Orion SDK is the right way to go. It's possible by invoking the swis verb Acknowledge over the "Orion.AlertStatus" entity.

          The parameters are: AlertDefID, ObjectType, ActiveObject and Notes (which you want to add to the alert when acknowledging).

          For acknowledging the alerts it's then possible e.g. to run the script (PowerShell, Perl, C# code).

          E.g. in Powershell it's like:

          Invoke-SwisVerb $swis "Orion.AlertStatus" "Acknowledge" @($alerts)
          
          

          where the $alerts is the array of the parameters.

           

          For more info, reffer to the Orion SDK documentation - Installers e.g. here: Re: Re: Orion SDK Information or if you don't want to execute the scripts (need to have installed the SDK), it's possible to use the REST/JSON interface

            • Re: Third Party Application Acknowledge SolarWinds Event
              cmwillis

              I looked into the JSON/REST interface a little bit, but it appears as if it is very limited in it's UPDATE functionality. The example code in the documentation just shows a poller being added with:

               

              https://localhost:17778/SolarWinds/InformationService/v3/Json/swis://tdannerdev.swdev.local/Orion/Orion.Pollers/PollerID=6
              
              

              Do you happen to know if there would be a way to specify the alert credentials listed in the alerts array you used in your Invoke example to perform an acknowledgement via JSON/REST that could also set the AcknowledgedBy parameter?

               

              I am attempting to avoid installing the SDK locally on Netcool to avoid potential implementation issues (this is a project I am researching for another enterprise, thus there may be certain limitations on what I can do), but it appears as though Invoke may be my best, and possible only, path forward.

              • Re: Third Party Application Acknowledge SolarWinds Event
                cmwillis

                I created the following PowerShell script:

                 

                param($hostname,$alertKey,$nodeAlias,$alertGroup)

                 

                # Load the SwisSnapin if not already loaded
                if (!(Get-PSSnapin | where {$_.Name -eq "SwisSnapin"})) {
                    Add-PSSnapin "SwisSnapin"
                }

                 

                #Prompt the user to enter credentials
                $credentials = Get-Credential
                $username = $credentials.GetNetworkCredential().domain + "\" + $credentials.GetNetworkCredential().username
                $password = $credentials.GetNetworkCredential().password | ConvertTo-SecureString -AsPlainText -Force
                $cred = New-Object -typename System.Management.Automation.PSCredential -argumentlist $username, $password

                 

                #Connect to SWIS
                $swis = Connect-Swis -host $hostname -cred $cred

                 

                #Acknowledge Alert
                $res = Invoke-SwisVerb $swis "Orion.AlertStatus" "Acknowledge" @($alertKey,$nodeAlias,$alertGroup)

                 

                As I said in my original post:

                @AlertKey = SolarWinds ${PropertyValue}

                @NodeAlias = SolarWinds ${NodeID}

                @AlertGroup = SolarWinds ${NetObjectType}

                 

                Whenever I run the script I get the error "Invoke-SwisVerb : Verb Orion.AlertStatus.Acknowledge cannot unpackage parameter 0"

                 

                Any idea what would cause this?

              • Re: Third Party Application Acknowledge SolarWinds Event
                cmwillis


                Okay, I was able to get the "Invoke-SwisVerb : Verb Orion.AlertStatus.Acknowledge cannot unpackage parameter 0" error to go away by modifying the array to be:

                 

                $res = Invoke-SwisVerb $swis "Orion.AlertStatus" "Acknowledge" @(,$alertKey,$nodeAlias,$alertGroup)

                 

                I'm not sure why that got rid of the error, but I saw it done in one of the examples I was looking at earlier so I figured I would try it.

                 

                The current issue is that the alert just simply doesn't get acknowledged after running the script. Any idea what I am doing wrong?

                  • Re: Re: Third Party Application Acknowledge SolarWinds Event
                    derhally

                    I'm not sure why that is not working, you can do the following for now.

                     

                    if (!(Get-PSSnapin -Name "SwisSnapin" -ErrorAction SilentlyContinue))
                    {
                      Add-PSSnapin SwisSnapin -ErrorAction SilentlyContinue
                    }
                    
                    $swisTarget = 'localhost'
                    $username="admin"
                    $password=""
                    
                    
                    $swis = Connect-Swis -host $swisTarget -UserName $username -Password $password
                    
                    #Get the first alert that isn't acknowledged
                    $query = "SELECT TOP 1 AlertDefID AS DefinitionId, ActiveObject AS ObjectId, ObjectType FROM Orion.AlertStatus WHERE Acknowledged = 0"
                    
                    $queryResult = Get-SwisData $swis $query
                    
                    ForEach ($ActiveAlert in $queryResult)
                    {
                      [xml]$invokeResult
                        $invokeResult = Invoke-SwisVerb $swis "Orion.AlertStatus" "Acknowledge" @(([xml]@(
                           "<ArrayOfAlerto xmlns='http://schemas.solarwinds.com/2008/Orion'>",
                           [string](%{
                             "<AlertInfo><DefinitionId>$($ActiveAlert.DefinitionId)</DefinitionId><ObjectType>$($ActiveAlert.ObjectType)</ObjectType><ObjectId>$($ActiveAlert.ObjectId)</ObjectId></AlertInfo>"
                             }
                           ),
                           "</ArrayOfAlertInfo>"
                        )).DocumentElement)
                    
                    
                      $result = $invokeResult.ChildNodes.Value
                    
                      Write-Host "Acknowledge events returned $result"
                    }
                    
                    
                    
                    
                    
                    
                    
                    
                  • Re: Third Party Application Acknowledge SolarWinds Event
                    cmwillis

                    Okay derhally, I was able to get the script working using the xml portion of your solution. This led me to the following code:

                     

                    param($swisTarget,$alertKey,$nodeAlias,$alertGroup)
                    
                    
                    if (!(Get-PSSnapin -Name "SwisSnapin" -ErrorAction SilentlyContinue))
                    {
                      Add-PSSnapin SwisSnapin -ErrorAction SilentlyContinue
                    }
                    
                    
                    $username = "admin"
                    $password = ""
                    
                    
                    $swis = Connect-Swis -host $swisTarget -UserName $username -Password $password
                    
                    [xml]$invokeResult 
                      $invokeResult = Invoke-SwisVerb $swis "Orion.AlertStatus" "Acknowledge" @(([xml]@( 
                        "<ArrayOfAlertInfo xmlns='http://schemas.solarwinds.com/2008/Orion'>",  
                        [string](%{ 
                          "<AlertInfo><DefinitionId>$($alertKey)</DefinitionId><ObjectType>$($alertGroup)</ObjectType><ObjectId>$($nodeAlias)</ObjectId></AlertInfo>" 
                          }
                        ),
                        "</ArrayOfAlertInfo>" 
                      )).DocumentElement)
                    
                    
                    $result = $invokeResult.ChildNodes.Value 
                    
                    Write-Host "Acknowledge events returned $result"
                    
                    
                    
                    
                    
                    
                    
                    
                    
                    
                    

                     

                    My current issue is setting the "AcknowledgedBy" column in the database. I tried using the xml tag <AcknowledgedBy>, but that still resulted in "admin" being the acknowledger even though I specified a different username.

                     

                    Any idea how to accomplish this?

                      • Re: Third Party Application Acknowledge SolarWinds Event
                        derhally

                        Currently it is no possible to do that with this verb.   This verb doesn't support taking that as an argument.   The arguments are as follows

                         

                        • DefinitionId
                        • ObjectType
                        • ObjectId
                        • Notes

                         

                        If you wish, you can use the Notes value to store the user who acknowledged the alert.

                          • Re: Third Party Application Acknowledge SolarWinds Event
                            cmwillis

                            We would like to try to avoid using the notes field if at all possible. Do you happen to know if there is a way to manually set the AcknowledgedBy value using a SQL command? Is this method even recommended?

                            • Re: Re: Third Party Application Acknowledge SolarWinds Event
                              ouberlord

                              Can you expand on the usage of ObjectId?  I assume it is analogous to Orion.AlertStatus.ActiveObject, but I can't get it to work.

                               

                              I intend for this script to address a series of actions for a particular type of alert that occurs on a particular set of nodes.  My code looks like the following, with the variables passed to the script as parameters:

                              [xml]$invokeResult 
                                $invokeResult = Invoke-SwisVerb $SWIS "Orion.AlertStatus" "Acknowledge" @(([xml]@( 
                                  "<ArrayOfAlertInfo xmlns='http://schemas.solarwinds.com/2008/Orion'>",   
                                  [string](%{ 
                                    "<AlertInfo><DefinitionId>$($AlertDefID)</DefinitionId><ObjectType>$("APM: Application")</ObjectType><ObjectId>$($NodeID)</ObjectId><Notes>$("Note goes here.")</Notes></AlertInfo>" 
                                    } 
                                  ), 
                                  "</ArrayOfAlertInfo>" 
                                )).DocumentElement)
                              

                               

                              I also assume that DefinitionId is the same as Orion.AlertStatus.AlertDefID and that ObjectType is the same as Orion.AlertStatus.ObjectType.

                                • Re: Re: Third Party Application Acknowledge SolarWinds Event
                                  tdanner

                                  You are correct about the correspondence of the DefinitionId->AlertDefID, ObjectType->ObjectType, and ObjectId->ActiveObject, but it doesn't look like your code really reflects that. You have hard-coded "APM: Application" as the ObjectType, but put $NodeID as the ObjectId. If the ObjectType is "APM: Application", the ObjectId will be an application ID, not a node ID.

                                    • Re: Re: Re: Third Party Application Acknowledge SolarWinds Event
                                      ouberlord

                                      Ah, right, kind of a mental typo there.  I'm testing the theory of using the API to Ack alerts based on a current Node Down alert I have.  However, I'm still struggling with the building of the proper XML array.

                                       

                                      In this test scenario, here is my alert's SWQL data:

                                      AlertDefID: abbe043e-a4e4-49d8-85fd-eb55ebb378dc

                                      ActiveObject: 378

                                      ObjectType: Node

                                       

                                      My thought is that this code should work:

                                      $AlertDefID = "abbe043e-a4e4-49d8-85fd-eb55ebb378dc"
                                      $NodeID = 378
                                      [xml]$invokeResult
                                        $invokeResult = Invoke-SwisVerb $SWIS "Orion.AlertStatus" "Acknowledge" @(([xml]@(
                                          "<ArrayOfAlertInfo xmlns='http://schemas.solarwinds.com/2008/Orion'>",   
                                          [string](%{
                                            "<AlertInfo><DefinitionId>$($AlertDefID)</DefinitionId><ObjectType>$("Node")</ObjectType><ObjectId>$($NodeID)</ObjectId><Notes>$("Testing.")</Notes></AlertInfo>"
                                            }
                                          ),
                                          "</ArrayOfAlertInfo>"
                                        )).DocumentElement)
                                      
                                      

                                       

                                      However, I get the following output:

                                      Cannot convert value "System.Xml.XmlElement" to type "System.Xml.XmlDocument". Error: "Data at the root level is invalid. Line 1, position 1."

                                       

                                      I have to be missing something painfully obvious.

                                      • Re: Re: Re: Third Party Application Acknowledge SolarWinds Event
                                        ouberlord

                                        Update:

                                         

                                        I've gotten it to work, but am still getting the XmlElement error.  Here is my code:

                                         

                                        $query = "SELECT TOP 1 AlertDefID AS DefinitionId, ActiveObject AS ObjectId, ObjectType FROM Orion.AlertStatus WHERE Acknowledged = 0 AND AlertDefID = 'abbe043e-a4e4-49d8-85fd-eb55ebb378dc'"
                                        $queryResult = Get-SwisData $swis $query
                                        $Notes = "Alert is being handled by an automatic Orion Recovery process."
                                        [xml]$invokeResult
                                          $invokeResult = Invoke-SwisVerb $swis "Orion.AlertStatus" "Acknowledge" @(([xml]@(
                                            "<ArrayOfAlertInfo xmlns='http://schemas.solarwinds.com/2008/Orion'>",
                                            [string](%{
                                              "<AlertInfo><DefinitionId>$($ActiveAlert.DefinitionId)</DefinitionId><ObjectType>$($ActiveAlert.ObjectType)</ObjectType><ObjectId>$($ActiveAlert.ObjectId)</ObjectId><Notes>$($Notes)</Notes></AlertInfo>"
                                              }
                                            ),
                                            "</ArrayOfAlertInfo>"
                                          )).DocumentElement)
                                        

                                         

                                        Technically this works; the alert is acked with the note I provided.  However, I still get this output:

                                         

                                        Cannot convert value "System.Xml.XmlElement" to type "System.Xml.XmlDocument". Error: "Data at the root level is invalid. Line 1, position 1."
                                        At line:1 char:19
                                        + [xml]$invokeResult <<<<
                                            + CategoryInfo          : NotSpecified: (:) [], RuntimeException
                                            + FullyQualifiedErrorId : RuntimeException
                                        

                                         

                                        How can I better structure the command so that it doesn't also generate this error?

                                          • Re: Re: Re: Re: Third Party Application Acknowledge SolarWinds Event
                                            tdanner

                                            Here's how I would do it:

                                             

                                            Add-PSSnapin SwisSnapin
                                            $swis = Connect-Swis latest-stable-builds -UserName admin -Password ""
                                            $queryResult = Get-SwisData $swis "SELECT TOP 1 AlertDefID AS DefinitionId, ActiveObject AS ObjectId, ObjectType FROM Orion.AlertStatus WHERE Acknowledged=0"
                                            $xmlString = "<ArrayOfAlertInfo xmlns='http://schemas.solarwinds.com/2008/Orion'>
                                                <AlertInfo>
                                                    <DefinitionId>$($queryResult.DefinitionId)</DefinitionId>
                                                    <ObjectType>$($queryResult.ObjectType)</ObjectType>
                                                    <ObjectId>$($queryResult.ObjectId)</ObjectId>
                                                </AlertInfo>
                                            </ArrayOfAlertInfo>"
                                            $xmlElement = ([xml]$xmlString).DocumentElement
                                            Invoke-SwisVerb $swis Orion.AlertStatus Acknowledge @( $xmlElement )
                                            
                                              • Re: Re: Re: Third Party Application Acknowledge SolarWinds Event
                                                ouberlord

                                                Eureka.  This is not only a method that works without throwing an error, but is much easier to read and understand.  I modified it by adding <Notes></Notes> and it works equally well if I populate it with $($Variable) or with $("Text goes here.").

                                                 

                                                Thank you for the help.  I was getting more and more lost by the day on this.

                                                  • Re: Third Party Application Acknowledge SolarWinds Event
                                                    johnny ringo

                                                    Did you ever get this to work for bulk alert acknowledgement?  I wrapped a for loop around the $xmlstring through the Invoke-Swisverb and the output returns multiple true statements, but I don't see alerts being acknowledged in the AlertStatus table.  In other parts of the SDK I didn't need a for loop to pass multiple URIs to the API, but I haven't had to use the xml strings before.

                                                      • Re: Third Party Application Acknowledge SolarWinds Event
                                                        tdanner

                                                        To acknowledge multiple alerts in one call, you need multiple <AlertInfo> elements inside the single <ArrayOfAlertInfo> element. Like this:

                                                         

                                                        Add-PSSnapin SwisSnapin  
                                                        $swis = Connect-Swis latest-stable-builds -UserName admin -Password ""  
                                                        $queryResult = Get-SwisData $swis "SELECT TOP 3 AlertDefID AS DefinitionId, ActiveObject AS ObjectId, ObjectType FROM Orion.AlertStatus WHERE Acknowledged=0"  
                                                        
                                                        $xmlString = "<ArrayOfAlertInfo xmlns='http://schemas.solarwinds.com/2008/Orion'>"
                                                        
                                                        foreach ($alert in $queryResult) {
                                                            $xmlString += "<AlertInfo>  
                                                                <DefinitionId>$($alert.DefinitionId)</DefinitionId>  
                                                                <ObjectType>$($alert.ObjectType)</ObjectType>  
                                                                <ObjectId>$($alert.ObjectId)</ObjectId>  
                                                            </AlertInfo>"
                                                        }
                                                        
                                                        $xmlString += "</ArrayOfAlertInfo>"
                                                        
                                                        $xmlElement = ([xml]$xmlString).DocumentElement  
                                                        Invoke-SwisVerb $swis Orion.AlertStatus Acknowledge @( $xmlElement )  
                                                        
                                        • Re: Third Party Application Acknowledge SolarWinds Event
                                          cmwillis

                                          Is there a way to get the Notes field to display in the web console under Home > Alerts?

                                          • Re: Third Party Application Acknowledge SolarWinds Event
                                            cmwillis

                                            I am also looking into the possibility of sending SNMP traps to SolarWinds to complete the acknowledgement. I was poking around in Trap Viewer, and it appeared that there were two methods to accomplish this:

                                             

                                            1. Using a regex to match the contents of the trap details
                                            2. Matching the SNMP trap against a series of OIDs that Trap Viewer has been preconfigured to look for

                                             

                                            My question is: once that match occurs within Trap Viewer, how do I then pass the AlertDefID received in the SNMP trap to a script that I am then triggering? Can Trap Viewer do this?